Sep 11 21:28:37.634926 ip-10-0-141-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 21:28:37.634939 ip-10-0-141-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 21:28:37.634948 ip-10-0-141-125 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 21:28:37.635259 ip-10-0-141-125 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 21:28:47.282862 ip-10-0-141-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 11 21:28:47.282877 ip-10-0-141-125 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 14e3413acbf3499f9ce72e1e9d137087 --
Sep 11 21:30:33.923027 ip-10-0-141-125 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:30:34.288510 ip-10-0-141-125 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:34.288510 ip-10-0-141-125 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:30:34.288510 ip-10-0-141-125 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:34.288510 ip-10-0-141-125 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:30:34.288510 ip-10-0-141-125 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:30:34.290133 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.290045    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:30:34.293926 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293911    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.293926 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293927    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293930    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293941    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293944    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293947    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293950    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293953    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293956    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293959    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293962    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293964    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293967    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293969    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293972    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293975    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293977    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293980    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293983    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293985    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.293998 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293988    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293991    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293994    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293996    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.293999    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294001    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294004    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294007    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294010    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294012    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294015    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294017    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294021    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294024    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294027    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294030    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294032    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294034    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294037    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294039    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.294451 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294042    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294044    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294047    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294051    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294054    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294057    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294060    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294062    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294065    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294068    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294070    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294073    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294076    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294079    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294083    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294086    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294089    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294093    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294096    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294098    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.294956 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294101    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294103    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294106    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294108    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294111    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294113    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294116    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294118    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294120    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294123    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294126    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294129    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294131    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294134    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294136    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294139    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294141    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294144    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294146    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294149    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.295474 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294151    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294155    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294157    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294160    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294163    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294166    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294169    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294171    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294174    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294177    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294179    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294182    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294185    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294187    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294189    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294192    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294196    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294199    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294202    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.295988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294905    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294912    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294916    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294919    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294921    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294924    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294927    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294930    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294933    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294935    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294938    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294941    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294944    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294946    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294949    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294951    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294954    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294957    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294960    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294964    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.296480 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294967    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294970    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294972    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294975    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294977    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294980    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294983    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294985    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294988    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294992    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294996    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.294999    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295002    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295005    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295007    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295010    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295012    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295015    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295018    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295021    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.296988 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295024    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295026    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295028    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295031    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295033    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295035    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295038    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295040    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295043    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295045    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295048    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295051    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295054    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295057    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295059    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295062    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295064    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295067    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295070    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295072    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.297467 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295075    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295077    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295080    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295082    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295084    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295087    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295089    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295092    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295094    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295096    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295100    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295102    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295105    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295107    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295110    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295112    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295115    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295117    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295120    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295122    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.298090 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295125    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295127    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295132    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295136    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295139    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295142    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295145    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295147    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295150    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295154    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295157    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295160    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295163    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295165    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295168    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295170    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295173    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295175    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295178    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.298590 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295596    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295602    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295605    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295608    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295612    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295615    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295617    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295620    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295623    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295626    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295628    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295631    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295633    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295638    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295642    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295645    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295648    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295651    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295654    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295657    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.299067 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295660    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295663    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295666    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295668    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295671    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295674    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295676    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295679    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295682    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295684    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295687    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295690    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295693    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295695    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295698    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295700    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295702    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295705    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295708    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295727    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.299561 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295730    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295732    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295735    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295737    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295740    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295746    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295748    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295751    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295753    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295756    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295758    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295761    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295763    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295766    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295768    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295771    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295773    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295787    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295791    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295794    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.300062 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295797    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295800    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295803    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295805    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295808    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295811    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295814    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295816    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295818    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295821    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295824    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295826    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295830    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295833    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295835    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295837    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295840    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295844    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295847    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295849    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.300573 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295852    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295854    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295857    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295860    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295864    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295867    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295869    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295873    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295879    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295882    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295885    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295888    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295890    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295893    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295896    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295899    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295901    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295904    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.295907    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.295970    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:30:34.301075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296007    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296015    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296025    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296034    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296041    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296046    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296062    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296066    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296070    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296074    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296077    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296080    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296083    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296086    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296089    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296092    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296095    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296100    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296103    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296106    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296110    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296114    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296117    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:30:34.301591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296120    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296123    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296127    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296129    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296132    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296135    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296138    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296141    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296150    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296153    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296155    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296158    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296161    2582 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296165    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296169    2582 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296172    2582 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296175    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296178    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296181    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296185    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296188    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296191    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296194    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296197    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296200    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:30:34.302182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296203    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296206    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296209    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296211    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296215    2582 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296220    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296223    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296226    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296229    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296232    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296235    2582 flags.go:64] FLAG: --help="false"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296238    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296241    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296243    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296246    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296250    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296253    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296256    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296259    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296261    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296264    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296268    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296271    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296274    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:30:34.302840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296277    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296280    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296283    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296286    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296289    2582 flags.go:64] FLAG: --lock-file=""
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296291    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296294    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296297    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296302    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296305    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296308    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296311    2582 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296315    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296319    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296321    2582 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296324    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296329    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296332    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296336    2582 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296339    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296342    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296345    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296348    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296351    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296354    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:30:34.303429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296357    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296364    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296367    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296370    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296373    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296376    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296381    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296384    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296387    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296390    2582 flags.go:64] FLAG: --port="10250"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296393    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296396    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0759b213076636059"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296399    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296402    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296405    2582 flags.go:64] FLAG: --register-node="true"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296408    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296412    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296414    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296417    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296421    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296425    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296428    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296431    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296434    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296436    2582 flags.go:64] FLAG: --runonce="false"
Sep 11 21:30:34.304100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296439    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296442    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296445    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296447    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296450    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296453    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296456    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296459    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296462    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296465    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296467    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296470    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296473    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296476    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296480    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296486    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296489    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296492    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296496    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296499    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296502    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296505    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296508    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296511    2582 flags.go:64] FLAG: --v="2"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296514    2582 flags.go:64] FLAG: --version="false"
Sep 11 21:30:34.304695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296518    2582 flags.go:64] FLAG: --vmodule=""
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296523    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.296526    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296647    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296650    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296653    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296656    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296659    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296662    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296665    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296668    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296671    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296675    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296679    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296682    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296685    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296687    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296690    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296692    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.305319 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296695    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296698    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296707    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296723    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296725    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296728    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296731    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296733    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296736    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296738    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296741    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296744    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296747    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296749    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296753    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296756    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296758    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296761    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296763    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296766    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.305815 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296768    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296771    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296774    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296776    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296779    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296781    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296784    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296787    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296789    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296791    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296794    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296796    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296799    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296802    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296805    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296813    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296816    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296818    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296821    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296824    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.306318 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296826    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296835    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296837    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296840    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296842    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296845    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296849    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296852    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296856    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296859    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296861    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296864    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296867    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296870    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296873    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296876    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296878    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296881    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296883    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.306817 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296886    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296889    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296891    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296894    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296896    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296898    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296901    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296905    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296908    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296917    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296921    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296924    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296926    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296929    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296932    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296934    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296937    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296940    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296942    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296946    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.307314 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296948    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296951    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296954    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.296957    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.297652    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.305988    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.306006    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.306810    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.306818    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306894    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306899    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306902    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306905    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306908    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306911    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.307882 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306914    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306916    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306919    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306922    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306924    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306927    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306931    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306935    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306938    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306940    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306943    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306945    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306948    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306951    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306953    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306956    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306958    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306961    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306963    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.308260 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306966    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306968    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306971    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306974    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306983    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306986    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306989    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306991    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306994    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306996    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.306999    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307001    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307004    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307006    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307009    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307013    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307017    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307020    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307023    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307026    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.308735 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307028    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307031    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307033    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307036    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307038    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307041    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307044    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307046    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307049    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307052    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307054    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307057    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307059    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307062    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307065    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307067    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307070    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307078    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307081    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307083    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.309215 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307086    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307088    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307091    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307093    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307096    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307098    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307100    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307103    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307105    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307108    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307110    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307113    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307115    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307118    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307120    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307123    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307125    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307128    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307130    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307132    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.309864 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307135    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307139    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307142    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307145    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307148    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307151    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307154    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307157    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307159    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307162    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307170    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307172    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307175    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307177    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.307182    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:34.310619 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307334    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307339    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307342    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307345    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307347    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307350    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307352    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307355    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307358    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307360    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307362    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307365    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307368    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307370    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307373    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307375    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307378    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307380    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307384    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:30:34.311027 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307388    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307392    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307395    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307398    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307401    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307404    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307407    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307410    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307413    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307426    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307429    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307431    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307434    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307436    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307439    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307441    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307444    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307447    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307449    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:30:34.311482 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307452    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307454    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307457    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307459    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307462    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307464    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307467    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307469    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307472    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307474    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307476    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307479    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307482    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307484    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307487    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307489    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307492    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307495    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307497    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307500    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:30:34.311954 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307502    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307505    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307507    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307515    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307518    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307520    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307523    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307525    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307527    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307530    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307532    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307535    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307537    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307539    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307542    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307544    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307547    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307549    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307552    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307554    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:30:34.312430 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307556    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307559    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307562    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307564    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307567    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307569    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307572    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307574    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307577    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307580    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307584    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307587    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307590    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307592    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307595    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307598    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307606    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307609    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307612    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307614    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:30:34.312924 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:34.307617    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.307622    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.307789    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.310766    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.311682    2582 server.go:1019] "Starting client certificate rotation"
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.311755    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:30:34.313424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.312705    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:30:34.314380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.314367    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:30:34.318941 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.318923    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:30:34.331250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.331233    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:30:34.334966 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.334950    2582 log.go:25] "Validated CRI v1 image API"
Sep 11 21:30:34.335765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.335741    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:30:34.338290 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.338272    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8c0fa43c-8a0c-4fc8-a732-21e591e1b1a1:/dev/nvme0n1p4 f18174dd-b287-4989-aee0-25b46b89b49c:/dev/nvme0n1p3]
Sep 11 21:30:34.338357 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.338290    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:30:34.344434 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.344312    2582 manager.go:216] Machine: {Timestamp:2026-09-11 21:30:34.343240066 +0000 UTC m=+0.306264705 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100301 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e1896bcdd0580d378f58d885ac40 SystemUUID:ec27e189-6bcd-d058-0d37-8f58d885ac40 BootID:14e3413a-cbf3-499f-9ce7-2e1e9d137087 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:74:8b:cd:ad:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:74:8b:cd:ad:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:c3:87:ad:c8:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:30:34.344434 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.344423    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:30:34.344595 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.344504    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:30:34.345435 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.345416    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:30:34.345681 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.345657    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:30:34.345891 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.345697    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:30:34.346529 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.346518    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:30:34.346591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.346534    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:30:34.346591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.346544    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:30:34.346591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.346562    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:30:34.348264 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.348251    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:30:34.352482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352468    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:30:34.352638 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352626    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:30:34.352709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352663    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:30:34.352709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352685    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:30:34.352709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352700    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:30:34.352860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352730    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:30:34.352860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.352754    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:30:34.353277 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.353262    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:30:34.353347 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.353288    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:30:34.355307 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.355289    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:30:34.358526 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.358507    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:30:34.358590 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.358554    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:30:34.360773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360756    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360780    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360789    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360797    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360808    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360817    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360835    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360842    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:30:34.360850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360852    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:30:34.361051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360861    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:30:34.361051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360873    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:30:34.361051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.360881    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:30:34.361549 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.361537    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:30:34.361549 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.361549    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:30:34.365295 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.365279    2582 server.go:1268] "Started kubelet"
Sep 11 21:30:34.365457 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.365372    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:30:34.365564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.365369    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:30:34.365564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.365559    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:30:34.366248 ip-10-0-141-125 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:30:34.367693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.367009    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 21:30:34.367693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.367324    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:30:34.367693 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.367487    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:30:34.367693 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.367489    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:30:34.368015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.368003    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:30:34.370210 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.370192    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:30:34.370849 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.369913    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-125.ec2.internal.18d46192a0e28ad8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-125.ec2.internal,UID:ip-10-0-141-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-125.ec2.internal,},FirstTimestamp:2026-09-11 21:30:34.365250264 +0000 UTC m=+0.328274904,LastTimestamp:2026-09-11 21:30:34.365250264 +0000 UTC m=+0.328274904,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-125.ec2.internal,}"
Sep 11 21:30:34.374015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.374000    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:30:34.374668 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.374641    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:30:34.374887 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.374868    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.375150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375134    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:30:34.375213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375151    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:30:34.375213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375162    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:30:34.375292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375279    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:30:34.375292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375290    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:30:34.375473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375459    2582 factory.go:55] Registering systemd factory
Sep 11 21:30:34.375507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375494    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:30:34.375751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375736    2582 factory.go:153] Registering CRI-O factory
Sep 11 21:30:34.375821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375753    2582 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:30:34.375821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375815    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:30:34.375908 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375835    2582 factory.go:103] Registering Raw factory
Sep 11 21:30:34.375908 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.375850    2582 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:30:34.376213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.376200    2582 manager.go:349] Starting recovery of all containers
Sep 11 21:30:34.376990 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.376907    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 21:30:34.377096 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.377060    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 21:30:34.386475 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.386308    2582 manager.go:354] Recovery completed
Sep 11 21:30:34.392666 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.392649    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.394527 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.394495    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.394601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.394544    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.394601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.394555    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.395194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.395176    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:30:34.395194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.395194    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:30:34.395310 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.395212    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:30:34.395310 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.395194    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.396004 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.395982    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.396052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.396014    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.396052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.396025    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.397331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.397315    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fm9rt"
Sep 11 21:30:34.397402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.397358    2582 policy_none.go:49] "None policy: Start"
Sep 11 21:30:34.397402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.397376    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:30:34.397402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.397390    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:30:34.399271 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.399258    2582 policy_none.go:47] "Start"
Sep 11 21:30:34.404173 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.404153    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fm9rt"
Sep 11 21:30:34.405997 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.405921    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-125.ec2.internal.18d46192a2a14888  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-125.ec2.internal,UID:ip-10-0-141-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-125.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-125.ec2.internal,},FirstTimestamp:2026-09-11 21:30:34.39452788 +0000 UTC m=+0.357552527,LastTimestamp:2026-09-11 21:30:34.39452788 +0000 UTC m=+0.357552527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-125.ec2.internal,}"
Sep 11 21:30:34.408621 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.408560    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-125.ec2.internal.18d46192a2a19c99  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-125.ec2.internal,UID:ip-10-0-141-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-125.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-125.ec2.internal,},FirstTimestamp:2026-09-11 21:30:34.394549401 +0000 UTC m=+0.357574041,LastTimestamp:2026-09-11 21:30:34.394549401 +0000 UTC m=+0.357574041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-125.ec2.internal,}"
Sep 11 21:30:34.437321 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437307    2582 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.437350    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437365    2582 server.go:87] "Starting device plugin registration server"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437669    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437679    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437794    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437869    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.437878    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.438423    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.438474    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.446551    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.447912    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.447954    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.447977    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.448021    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 21:30:34.459963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.449904    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:34.538408 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.538376    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.539615 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.539529    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.539615 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.539561    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.539615 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.539577    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.539615 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.539603    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.546706 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.546690    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.546792 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.546723    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-125.ec2.internal\": node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.548922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.548903    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"]
Sep 11 21:30:34.548993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.548982    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.555469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.555452    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.555974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.555964    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.556020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.555979    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.559894 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.559880    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.560024 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.560008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.560061 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.560042    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.561081 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561065    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.561158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561096    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.561158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561111    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.561764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561748    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.561830 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561775    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.561830 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.561786    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.563955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.563941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.564029 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.563966    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:34.570323 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.570305    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:34.570418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.570338    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:34.570418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.570352    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:34.576873 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.576855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.576955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.576880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.576955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.576904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91a6bb631a6b60831cd09f18b64e9aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-125.ec2.internal\" (UID: \"91a6bb631a6b60831cd09f18b64e9aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.580325 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.580311    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-125.ec2.internal\" not found" node="ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.586144 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.586130    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-125.ec2.internal\" not found" node="ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.588615 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.588600    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.677552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.677552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.677707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91a6bb631a6b60831cd09f18b64e9aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-125.ec2.internal\" (UID: \"91a6bb631a6b60831cd09f18b64e9aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.677707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.677707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4046af0c0e0cdb8e9f0c2742b75ed38a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal\" (UID: \"4046af0c0e0cdb8e9f0c2742b75ed38a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.677707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.677638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91a6bb631a6b60831cd09f18b64e9aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-125.ec2.internal\" (UID: \"91a6bb631a6b60831cd09f18b64e9aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.689443 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.689424    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.790457 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.790386    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.883120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.883085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.891184 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.891157    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:34.891253 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:34.891184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:34.991591 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:34.991551    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.092133 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.092057    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.192622 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.192588    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.293132 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.293096    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.311265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.311240    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:30:35.311419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.311401    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:30:35.393506 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.393443    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.395859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.395833    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:35.406821 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:35.406785    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4046af0c0e0cdb8e9f0c2742b75ed38a.slice/crio-4883e089759dfa98254d72168845e41d57031f5509fb683ea004e4db157ba546 WatchSource:0}: Error finding container 4883e089759dfa98254d72168845e41d57031f5509fb683ea004e4db157ba546: Status 404 returned error can't find the container with id 4883e089759dfa98254d72168845e41d57031f5509fb683ea004e4db157ba546
Sep 11 21:30:35.407040 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:35.407021    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a6bb631a6b60831cd09f18b64e9aa6.slice/crio-51dd049964c954316d5aacfc1d768a207dd492a80e4a8395e25dc5f847efc498 WatchSource:0}: Error finding container 51dd049964c954316d5aacfc1d768a207dd492a80e4a8395e25dc5f847efc498: Status 404 returned error can't find the container with id 51dd049964c954316d5aacfc1d768a207dd492a80e4a8395e25dc5f847efc498
Sep 11 21:30:35.407113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.407063    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:25:35 +0000 UTC" deadline="2028-05-05 16:33:05.775347429 +0000 UTC"
Sep 11 21:30:35.407113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.407088    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14443h2m30.368263337s"
Sep 11 21:30:35.411311 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.411282    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:30:35.452149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.452094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal" event={"ID":"91a6bb631a6b60831cd09f18b64e9aa6","Type":"ContainerStarted","Data":"51dd049964c954316d5aacfc1d768a207dd492a80e4a8395e25dc5f847efc498"}
Sep 11 21:30:35.453117 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.453099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal" event={"ID":"4046af0c0e0cdb8e9f0c2742b75ed38a","Type":"ContainerStarted","Data":"4883e089759dfa98254d72168845e41d57031f5509fb683ea004e4db157ba546"}
Sep 11 21:30:35.493587 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.493539    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.571781 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.571743    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:30:35.573096 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.573081    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:30:35.573174 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.573112    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:30:35.573174 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.573123    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:30:35.594625 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.594595    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.695129 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.695053    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.696969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.696954    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:35.795965 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.795936    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.896750 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:35.896706    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-125.ec2.internal\" not found"
Sep 11 21:30:35.919026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.919001    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:35.975864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.975782    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:35.984468 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.984439    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:30:35.985467 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:35.985449    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal"
Sep 11 21:30:36.000038 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.000019    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:30:36.354391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.354305    2582 apiserver.go:52] "Watching apiserver"
Sep 11 21:30:36.374380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.374346    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:30:36.374636 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.374618    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:30:36.376897 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.376857    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw","openshift-cluster-node-tuning-operator/tuned-qgbrm","openshift-dns/node-resolver-h4xrb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal","openshift-network-diagnostics/network-check-target-w7md7","openshift-network-operator/iptables-alerter-qjn7j","openshift-ovn-kubernetes/ovnkube-node-trnpl","kube-system/global-pull-secret-syncer-shbrv","kube-system/konnectivity-agent-6pfdg","kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal","openshift-image-registry/node-ca-bqqsd","openshift-multus/multus-6c5gv","openshift-multus/multus-additional-cni-plugins-gnknv","openshift-multus/network-metrics-daemon-x7x2x"]
Sep 11 21:30:36.380110 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.380084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.382349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.382331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.383494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.383472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rgmgg\""
Sep 11 21:30:36.383598 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.383505    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:30:36.383762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.383748    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.383894 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.383758    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:30:36.383973 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.383955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.384124 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.384107    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:30:36.385016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.384994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.385217 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.385196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sx64x\""
Sep 11 21:30:36.385398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.385354    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.385608 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.385590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.387587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.387564    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.387685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.387617    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rdv4w\""
Sep 11 21:30:36.387990 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.387970    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.388185 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.388167    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.389068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.389045    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:30:36.390605 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.390583    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:30:36.390776 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.390754    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.391535 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.390947    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tbpbt\""
Sep 11 21:30:36.391535 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.391069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:36.391535 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.391070    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.391535 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.391141    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:36.394194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.393949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.396305 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.396281    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.396416 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.396368    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5j6pv\""
Sep 11 21:30:36.396530 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.396515    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.396639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.396621    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:30:36.397207 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.397188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.399824 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.399800    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:30:36.399993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.399958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.400104 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.400041    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:30:36.400104 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.400054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.400207 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.400114    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:36.400919 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.400898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:30:36.401151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.401129    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.401410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.401391    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2fcl\""
Sep 11 21:30:36.401635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.401617    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:30:36.402654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.402636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.404864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.404845    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:30:36.406064 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.406005    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5s2gr\""
Sep 11 21:30:36.407651 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.406076    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:30:36.407651 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.406416    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.407651 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.406500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:36.407959 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.407935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.411390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.411373    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:30:36.412219 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.412199    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mdkhs\""
Sep 11 21:30:36.412317 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.412247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.412544 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.412524    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:30:36.412922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.412734    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:30:36.414688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.414453    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fbdtq\""
Sep 11 21:30:36.414688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.414498    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:30:36.417082 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.416568    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:30:36.417551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.417532    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mfrdq"
Sep 11 21:30:36.424158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.424133    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mfrdq"
Sep 11 21:30:36.476784 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.476752    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:30:36.489780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-run\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.489780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-tmp\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-log-socket\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-multus-certs\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-slash\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac767800-1710-4e52-a859-2b3a9047369c-host\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-socket-dir-parent\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-lib-modules\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5971bb-454c-4913-b734-a524a351bb66-tmp-dir\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.489991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.489997    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-etc-kubernetes\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-host\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlh5\" (UniqueName: \"kubernetes.io/projected/8b469ddc-2dbc-43b7-8719-8cc428baefef-kube-api-access-8tlh5\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490080    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k69\" (UniqueName: \"kubernetes.io/projected/cc5971bb-454c-4913-b734-a524a351bb66-kube-api-access-68k69\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-registration-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-bin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-cnibin\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-ovn\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-device-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-conf-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202e2ea8-7802-492c-aa6d-24c814a08662-host-slash\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-modprobe-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvsmf\" (UniqueName: \"kubernetes.io/projected/b1a50d4a-fa78-4a02-8219-4dba86f42965-kube-api-access-dvsmf\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.490423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-multus\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-var-lib-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-node-log\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-cni-binary-copy\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzlk\" (UniqueName: \"kubernetes.io/projected/584ee74d-2923-46fe-a902-cfffb43f7d32-kube-api-access-8tzlk\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-etc-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-sys-fs\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89t97\" (UniqueName: \"kubernetes.io/projected/339eabf9-164e-4e88-b61e-f7ba222ec501-kube-api-access-89t97\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wq4\" (UniqueName: \"kubernetes.io/projected/4dd4f841-9875-4dbd-be36-56e9a9bd205d-kube-api-access-d6wq4\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-k8s-cni-cncf-io\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.490961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-system-cni-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-sys\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-var-lib-kubelet\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-netns\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-systemd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-script-lib\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-kubelet-config\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-conf\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-systemd\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-tuned\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nc2\" (UniqueName: \"kubernetes.io/projected/ac767800-1710-4e52-a859-2b3a9047369c-kube-api-access-d4nc2\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-daemon-config\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovn-node-metrics-cert\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-konnectivity-ca\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5971bb-454c-4913-b734-a524a351bb66-hosts-file\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491473    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-socket-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.491864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtjc\" (UniqueName: \"kubernetes.io/projected/202e2ea8-7802-492c-aa6d-24c814a08662-kube-api-access-sdtjc\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-bin\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac767800-1710-4e52-a859-2b3a9047369c-serviceca\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491586    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-system-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-netns\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-hostroot\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-kubernetes\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/202e2ea8-7802-492c-aa6d-24c814a08662-iptables-alerter-script\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-kubelet\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysconfig\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-systemd-units\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-kubelet\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-netd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.492499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.491995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-config\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.493059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.492019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-env-overrides\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.493059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.492035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-cnibin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.493059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.492052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-dbus\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.493059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.492067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-agent-certs\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.493059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.492088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ww5\" (UniqueName: \"kubernetes.io/projected/b85435a6-55d3-4b79-997c-15a41ae02555-kube-api-access-88ww5\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.593047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-tmp\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-log-socket\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-multus-certs\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-slash\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac767800-1710-4e52-a859-2b3a9047369c-host\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-socket-dir-parent\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-log-socket\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-lib-modules\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.593250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5971bb-454c-4913-b734-a524a351bb66-tmp-dir\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-etc-kubernetes\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-host\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlh5\" (UniqueName: \"kubernetes.io/projected/8b469ddc-2dbc-43b7-8719-8cc428baefef-kube-api-access-8tlh5\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k69\" (UniqueName: \"kubernetes.io/projected/cc5971bb-454c-4913-b734-a524a351bb66-kube-api-access-68k69\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-lib-modules\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-registration-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593434    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-slash\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593483    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-registration-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-bin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-multus-certs\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-cnibin\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-ovn\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.593672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-device-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-host\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-conf-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202e2ea8-7802-492c-aa6d-24c814a08662-host-slash\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-modprobe-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvsmf\" (UniqueName: \"kubernetes.io/projected/b1a50d4a-fa78-4a02-8219-4dba86f42965-kube-api-access-dvsmf\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-multus\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.593982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-etc-selinux\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-bin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc5971bb-454c-4913-b734-a524a351bb66-tmp-dir\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-cnibin\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-var-lib-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-node-log\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.594531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-device-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-cni-binary-copy\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzlk\" (UniqueName: \"kubernetes.io/projected/584ee74d-2923-46fe-a902-cfffb43f7d32-kube-api-access-8tzlk\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594365    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-etc-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-etc-kubernetes\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594444    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-sys-fs\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89t97\" (UniqueName: \"kubernetes.io/projected/339eabf9-164e-4e88-b61e-f7ba222ec501-kube-api-access-89t97\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-sys-fs\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.593577    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wq4\" (UniqueName: \"kubernetes.io/projected/4dd4f841-9875-4dbd-be36-56e9a9bd205d-kube-api-access-d6wq4\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-k8s-cni-cncf-io\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-system-cni-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.595381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.594662    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:37.094637974 +0000 UTC m=+3.057662604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594703    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-sys\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-var-lib-kubelet\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-netns\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-systemd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-script-lib\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594853    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-kubelet-config\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-conf-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-conf\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-systemd\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-tuned\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nc2\" (UniqueName: \"kubernetes.io/projected/ac767800-1710-4e52-a859-2b3a9047369c-kube-api-access-d4nc2\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.594999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-conf\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-daemon-config\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:36.596244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovn-node-metrics-cert\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-konnectivity-ca\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5971bb-454c-4913-b734-a524a351bb66-hosts-file\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-socket-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtjc\" (UniqueName: \"kubernetes.io/projected/202e2ea8-7802-492c-aa6d-24c814a08662-kube-api-access-sdtjc\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-bin\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac767800-1710-4e52-a859-2b3a9047369c-serviceca\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-system-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-netns\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-hostroot\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-kubernetes\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac767800-1710-4e52-a859-2b3a9047369c-host\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/202e2ea8-7802-492c-aa6d-24c814a08662-iptables-alerter-script\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-var-lib-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-kubelet\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysconfig\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.597036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-node-log\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-systemd-units\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-kubelet\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-netd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-config\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-env-overrides\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-cnibin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-dbus\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-daemon-config\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-agent-certs\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ww5\" (UniqueName: \"kubernetes.io/projected/b85435a6-55d3-4b79-997c-15a41ae02555-kube-api-access-88ww5\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-run\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-k8s-cni-cncf-io\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-kubelet-config\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-kubernetes\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.597936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.595984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysconfig\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-systemd-units\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-kubelet\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596183    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-netd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-cni-bin\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc5971bb-454c-4913-b734-a524a351bb66-hosts-file\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-binary-copy\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-kubelet\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596861    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-modprobe-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-config\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.596945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/339eabf9-164e-4e88-b61e-f7ba222ec501-socket-dir\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.596960    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.597012    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:30:37.096996475 +0000 UTC m=+3.060021126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-konnectivity-ca\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-systemd\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.598886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-etc-openvswitch\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-run-netns\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/202e2ea8-7802-492c-aa6d-24c814a08662-host-slash\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-system-cni-dir\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-ovn\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-host-var-lib-cni-multus\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac767800-1710-4e52-a859-2b3a9047369c-serviceca\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85435a6-55d3-4b79-997c-15a41ae02555-system-cni-dir\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-sysctl-d\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-sys\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-var-lib-kubelet\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-netns\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-env-overrides\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/202e2ea8-7802-492c-aa6d-24c814a08662-iptables-alerter-script\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.597976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-run-systemd\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-cnibin\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b1a50d4a-fa78-4a02-8219-4dba86f42965-host-run-ovn-kubernetes\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-multus-socket-dir-parent\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.599732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ee5ac432-0461-4ede-ac81-49fa12097464-dbus\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:36.600736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovnkube-script-lib\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.600736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.598608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/584ee74d-2923-46fe-a902-cfffb43f7d32-cni-binary-copy\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.600736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.599291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-tmp\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.600736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.599359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b1a50d4a-fa78-4a02-8219-4dba86f42965-ovn-node-metrics-cert\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.600736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.600070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b469ddc-2dbc-43b7-8719-8cc428baefef-etc-tuned\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.601044 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.600929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744-agent-certs\") pod \"konnectivity-agent-6pfdg\" (UID: \"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744\") " pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.602595 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.602571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/584ee74d-2923-46fe-a902-cfffb43f7d32-hostroot\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.602887 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.602799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b469ddc-2dbc-43b7-8719-8cc428baefef-run\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.603097 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.603075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.603704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.603679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b85435a6-55d3-4b79-997c-15a41ae02555-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.607406 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.607333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlh5\" (UniqueName: \"kubernetes.io/projected/8b469ddc-2dbc-43b7-8719-8cc428baefef-kube-api-access-8tlh5\") pod \"tuned-qgbrm\" (UID: \"8b469ddc-2dbc-43b7-8719-8cc428baefef\") " pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.608170 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.608150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k69\" (UniqueName: \"kubernetes.io/projected/cc5971bb-454c-4913-b734-a524a351bb66-kube-api-access-68k69\") pod \"node-resolver-h4xrb\" (UID: \"cc5971bb-454c-4913-b734-a524a351bb66\") " pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.615060 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.615035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wq4\" (UniqueName: \"kubernetes.io/projected/4dd4f841-9875-4dbd-be36-56e9a9bd205d-kube-api-access-d6wq4\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:36.618443 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.618416    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:36.618443 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.618444    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:36.618763 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.618458    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:36.618763 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:36.618541    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:37.11852179 +0000 UTC m=+3.081546426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:36.620546 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.620523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvsmf\" (UniqueName: \"kubernetes.io/projected/b1a50d4a-fa78-4a02-8219-4dba86f42965-kube-api-access-dvsmf\") pod \"ovnkube-node-trnpl\" (UID: \"b1a50d4a-fa78-4a02-8219-4dba86f42965\") " pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.624272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.624245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtjc\" (UniqueName: \"kubernetes.io/projected/202e2ea8-7802-492c-aa6d-24c814a08662-kube-api-access-sdtjc\") pod \"iptables-alerter-qjn7j\" (UID: \"202e2ea8-7802-492c-aa6d-24c814a08662\") " pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.624411 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.624386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ww5\" (UniqueName: \"kubernetes.io/projected/b85435a6-55d3-4b79-997c-15a41ae02555-kube-api-access-88ww5\") pod \"multus-additional-cni-plugins-gnknv\" (UID: \"b85435a6-55d3-4b79-997c-15a41ae02555\") " pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.626083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.626058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nc2\" (UniqueName: \"kubernetes.io/projected/ac767800-1710-4e52-a859-2b3a9047369c-kube-api-access-d4nc2\") pod \"node-ca-bqqsd\" (UID: \"ac767800-1710-4e52-a859-2b3a9047369c\") " pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.628705 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.628679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzlk\" (UniqueName: \"kubernetes.io/projected/584ee74d-2923-46fe-a902-cfffb43f7d32-kube-api-access-8tzlk\") pod \"multus-6c5gv\" (UID: \"584ee74d-2923-46fe-a902-cfffb43f7d32\") " pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:36.629198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.629175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89t97\" (UniqueName: \"kubernetes.io/projected/339eabf9-164e-4e88-b61e-f7ba222ec501-kube-api-access-89t97\") pod \"aws-ebs-csi-driver-node-wbfqw\" (UID: \"339eabf9-164e-4e88-b61e-f7ba222ec501\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.694832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.694795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gnknv"
Sep 11 21:30:36.702942 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.702917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qgbrm"
Sep 11 21:30:36.713954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.713929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4xrb"
Sep 11 21:30:36.720909 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.720883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw"
Sep 11 21:30:36.729896 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.729871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qjn7j"
Sep 11 21:30:36.739990 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.739967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:30:36.748138 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.748108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:36.756029 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.756009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bqqsd"
Sep 11 21:30:36.762071 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:36.762051    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6c5gv"
Sep 11 21:30:37.099520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.099415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:37.099520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.099489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:37.099802 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.099609    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:37.099802 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.099621    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:37.099802 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.099692    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:30:38.099675781 +0000 UTC m=+4.062700430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:37.099802 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.099713    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:38.099702729 +0000 UTC m=+4.062727363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:37.200448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.200411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:37.200613 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.200554    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:37.200613 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.200571    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:37.200613 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.200581    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:37.200768 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.200640    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:38.200621864 +0000 UTC m=+4.163646493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:37.427269 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.427150    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:25:37 +0000 UTC" deadline="2028-04-30 01:55:52.428362938 +0000 UTC"
Sep 11 21:30:37.427269 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.427214    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14308h25m15.001151941s"
Sep 11 21:30:37.448474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:37.448437    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:37.448646 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:37.448558    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:37.557033 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.557001    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202e2ea8_7802_492c_aa6d_24c814a08662.slice/crio-05e506756a4e31f6761d1fe7d12b7828832d46ed555060ce5e5eb29447cf141c WatchSource:0}: Error finding container 05e506756a4e31f6761d1fe7d12b7828832d46ed555060ce5e5eb29447cf141c: Status 404 returned error can't find the container with id 05e506756a4e31f6761d1fe7d12b7828832d46ed555060ce5e5eb29447cf141c
Sep 11 21:30:37.559239 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.559216    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b469ddc_2dbc_43b7_8719_8cc428baefef.slice/crio-e76a1d49a48e3eeb5c5a876507674cb56a34b2c9a662f221e6b7aca6a60eee97 WatchSource:0}: Error finding container e76a1d49a48e3eeb5c5a876507674cb56a34b2c9a662f221e6b7aca6a60eee97: Status 404 returned error can't find the container with id e76a1d49a48e3eeb5c5a876507674cb56a34b2c9a662f221e6b7aca6a60eee97
Sep 11 21:30:37.560787 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.560744    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a50d4a_fa78_4a02_8219_4dba86f42965.slice/crio-b990f0d9154826d9cd0f383ccea1f62cf73c68e32d9359456c1055cbadf5a78e WatchSource:0}: Error finding container b990f0d9154826d9cd0f383ccea1f62cf73c68e32d9359456c1055cbadf5a78e: Status 404 returned error can't find the container with id b990f0d9154826d9cd0f383ccea1f62cf73c68e32d9359456c1055cbadf5a78e
Sep 11 21:30:37.562306 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.562285    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85435a6_55d3_4b79_997c_15a41ae02555.slice/crio-a83692dafb85c99f192dee5b2fda31c1d85e4569707b98a9ebe9ee3954f43bca WatchSource:0}: Error finding container a83692dafb85c99f192dee5b2fda31c1d85e4569707b98a9ebe9ee3954f43bca: Status 404 returned error can't find the container with id a83692dafb85c99f192dee5b2fda31c1d85e4569707b98a9ebe9ee3954f43bca
Sep 11 21:30:37.563023 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.562999    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa94ba89_8fdd_4cd4_90d8_04ab4e2c7744.slice/crio-d8758f07776705d373388737e41f63a0e42187779f0a1aae76e62468c5d74c05 WatchSource:0}: Error finding container d8758f07776705d373388737e41f63a0e42187779f0a1aae76e62468c5d74c05: Status 404 returned error can't find the container with id d8758f07776705d373388737e41f63a0e42187779f0a1aae76e62468c5d74c05
Sep 11 21:30:37.564486 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.564357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac767800_1710_4e52_a859_2b3a9047369c.slice/crio-d1e7d5c44f5c70cfe4d5d165d288196aa65019f7e0e95934af8cbe18c69a721e WatchSource:0}: Error finding container d1e7d5c44f5c70cfe4d5d165d288196aa65019f7e0e95934af8cbe18c69a721e: Status 404 returned error can't find the container with id d1e7d5c44f5c70cfe4d5d165d288196aa65019f7e0e95934af8cbe18c69a721e
Sep 11 21:30:37.588550 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.588505    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339eabf9_164e_4e88_b61e_f7ba222ec501.slice/crio-cb795e207b69ba4b0c59bd0078e444f1767a556beaaa7ac942919337da6c4d9b WatchSource:0}: Error finding container cb795e207b69ba4b0c59bd0078e444f1767a556beaaa7ac942919337da6c4d9b: Status 404 returned error can't find the container with id cb795e207b69ba4b0c59bd0078e444f1767a556beaaa7ac942919337da6c4d9b
Sep 11 21:30:37.590149 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.589819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584ee74d_2923_46fe_a902_cfffb43f7d32.slice/crio-413240af5e91df5132d4313446b6282dbdd42f145451ac4699b04963c0f94b84 WatchSource:0}: Error finding container 413240af5e91df5132d4313446b6282dbdd42f145451ac4699b04963c0f94b84: Status 404 returned error can't find the container with id 413240af5e91df5132d4313446b6282dbdd42f145451ac4699b04963c0f94b84
Sep 11 21:30:37.590517 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:30:37.590495    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5971bb_454c_4913_b734_a524a351bb66.slice/crio-c0f968cc80c87c43169aa886fa88cbafcb38523af8d8cbfdbc49243aa1bc1551 WatchSource:0}: Error finding container c0f968cc80c87c43169aa886fa88cbafcb38523af8d8cbfdbc49243aa1bc1551: Status 404 returned error can't find the container with id c0f968cc80c87c43169aa886fa88cbafcb38523af8d8cbfdbc49243aa1bc1551
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.107362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.107429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.107584    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.107638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.107623366 +0000 UTC m=+6.070648008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.108024    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:38.108218 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.108073    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.108059997 +0000 UTC m=+6.071084645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:38.208520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.208482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:38.208722 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.208638    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:38.208722 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.208660    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:38.208722 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.208670    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:38.208722 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.208721    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:40.208704384 +0000 UTC m=+6.171729013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:38.427905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.427806    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:25:37 +0000 UTC" deadline="2028-05-19 21:42:57.563540494 +0000 UTC"
Sep 11 21:30:38.427905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.427852    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14784h12m19.135692923s"
Sep 11 21:30:38.451626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.451594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:38.451803 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.451741    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:38.452212 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.452189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:38.452329 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:38.452309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:38.479224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.478515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal" event={"ID":"91a6bb631a6b60831cd09f18b64e9aa6","Type":"ContainerStarted","Data":"085c01208bdccc9653a4d0456bafe2c2093a3daa88f60b12d7627d09a4b8430f"}
Sep 11 21:30:38.492094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.492040    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw" event={"ID":"339eabf9-164e-4e88-b61e-f7ba222ec501","Type":"ContainerStarted","Data":"cb795e207b69ba4b0c59bd0078e444f1767a556beaaa7ac942919337da6c4d9b"}
Sep 11 21:30:38.505644 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.505559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bqqsd" event={"ID":"ac767800-1710-4e52-a859-2b3a9047369c","Type":"ContainerStarted","Data":"d1e7d5c44f5c70cfe4d5d165d288196aa65019f7e0e95934af8cbe18c69a721e"}
Sep 11 21:30:38.515981 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.515942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6pfdg" event={"ID":"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744","Type":"ContainerStarted","Data":"d8758f07776705d373388737e41f63a0e42187779f0a1aae76e62468c5d74c05"}
Sep 11 21:30:38.526859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.526813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"b990f0d9154826d9cd0f383ccea1f62cf73c68e32d9359456c1055cbadf5a78e"}
Sep 11 21:30:38.548108 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.547991    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qjn7j" event={"ID":"202e2ea8-7802-492c-aa6d-24c814a08662","Type":"ContainerStarted","Data":"05e506756a4e31f6761d1fe7d12b7828832d46ed555060ce5e5eb29447cf141c"}
Sep 11 21:30:38.581563 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.581520    2582 generic.go:356] "Generic (PLEG): container finished" podID="4046af0c0e0cdb8e9f0c2742b75ed38a" containerID="662dcd7dd70e3de51bbcb443da417d75f301a97252f1873851566dd0ac814691" exitCode=0
Sep 11 21:30:38.581818 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.581651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal" event={"ID":"4046af0c0e0cdb8e9f0c2742b75ed38a","Type":"ContainerDied","Data":"662dcd7dd70e3de51bbcb443da417d75f301a97252f1873851566dd0ac814691"}
Sep 11 21:30:38.605103 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.604495    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-125.ec2.internal" podStartSLOduration=2.604469279 podStartE2EDuration="2.604469279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:38.497820524 +0000 UTC m=+4.460845169" watchObservedRunningTime="2026-09-11 21:30:38.604469279 +0000 UTC m=+4.567493936"
Sep 11 21:30:38.606659 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.606588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4xrb" event={"ID":"cc5971bb-454c-4913-b734-a524a351bb66","Type":"ContainerStarted","Data":"c0f968cc80c87c43169aa886fa88cbafcb38523af8d8cbfdbc49243aa1bc1551"}
Sep 11 21:30:38.634250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.634145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6c5gv" event={"ID":"584ee74d-2923-46fe-a902-cfffb43f7d32","Type":"ContainerStarted","Data":"413240af5e91df5132d4313446b6282dbdd42f145451ac4699b04963c0f94b84"}
Sep 11 21:30:38.646334 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.646297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerStarted","Data":"a83692dafb85c99f192dee5b2fda31c1d85e4569707b98a9ebe9ee3954f43bca"}
Sep 11 21:30:38.661201 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:38.661156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qgbrm" event={"ID":"8b469ddc-2dbc-43b7-8719-8cc428baefef","Type":"ContainerStarted","Data":"e76a1d49a48e3eeb5c5a876507674cb56a34b2c9a662f221e6b7aca6a60eee97"}
Sep 11 21:30:39.449561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:39.449024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:39.449561 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:39.449186    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:39.691950 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:39.691911    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal" event={"ID":"4046af0c0e0cdb8e9f0c2742b75ed38a","Type":"ContainerStarted","Data":"e65040ebffcfa653cb907ca8ff6eadcc164b5f1ddc2836d6a8aa8bde5bf39b3c"}
Sep 11 21:30:40.127595 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:40.127547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:40.127808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:40.127662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:40.127881 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.127816    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:40.127935 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.127883    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:44.127863798 +0000 UTC m=+10.090888436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:40.128325 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.128302    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:40.128433 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.128367    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:30:44.128349455 +0000 UTC m=+10.091374086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:40.229291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:40.228625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:40.229291 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.228826    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:40.229291 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.228850    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:40.229291 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.228865    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:40.229291 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.228929    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:44.228909396 +0000 UTC m=+10.191934038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:40.450039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:40.449943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:40.450532 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.450192    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:40.450625 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:40.450603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:40.451338 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:40.450830    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:41.448563 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:41.448510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:41.448766 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:41.448706    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:42.452250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:42.451971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:42.452809 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:42.452367    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:42.452809 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:42.452770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:42.452960 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:42.452908    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:43.449092 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:43.448967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:43.449260 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:43.449113    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:44.166463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:44.166422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:44.167364 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:44.166490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:44.167364 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.166651    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:44.167364 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.166716    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:30:52.166697774 +0000 UTC m=+18.129722421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:44.167364 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.166808    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:44.167364 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.166846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:52.166833833 +0000 UTC m=+18.129858479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:44.267477 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:44.267437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:44.267658 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.267646    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:44.267717 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.267669    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:44.267717 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.267682    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:44.267850 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.267742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:30:52.267724285 +0000 UTC m=+18.230748930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:44.455497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:44.455409    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:44.455668 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.455600    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:44.456082 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:44.456063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:44.456186 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:44.456161    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:45.448582 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:45.448540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:45.449042 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:45.448684    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:46.449259 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:46.449212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:46.449851 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:46.449289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:46.450015 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:46.449892    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:46.450015 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:46.449968    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:47.448734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:47.448676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:47.448903 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:47.448828    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:48.448583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:48.448544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:48.449051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:48.448599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:48.449051 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:48.448699    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:48.449051 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:48.448825    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:49.448463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:49.448383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:49.448616 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:49.448509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:50.449020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:50.448983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:50.449518 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:50.449125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:50.449518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:50.449180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:50.449518 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:50.449299    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:51.448754 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:51.448709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:51.448979 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:51.448837    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:52.234572 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:52.234533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:52.235030 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:52.234596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:52.235030 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.234689    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:52.235030 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.234709    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:52.235030 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.234778    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:08.234760024 +0000 UTC m=+34.197784665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:30:52.235030 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.234796    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:08.234787339 +0000 UTC m=+34.197811966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:30:52.335921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:52.335880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:52.336103 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.336071    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:30:52.336103 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.336102    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:30:52.336196 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.336116    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:52.336196 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.336178    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:08.336159313 +0000 UTC m=+34.299183963 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:30:52.448971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:52.448938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:52.449150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:52.448944    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:52.449150 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.449103    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:52.449266 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:52.449230    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:53.448950 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:53.448915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:53.449422 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:53.449059    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:54.449387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:54.449354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:54.449863 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:54.449436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:54.449863 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:54.449457    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:54.449863 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:54.449557    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:55.449128 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:55.449084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:55.449328 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:55.449225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:56.449034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.448996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:56.449518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.448998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:56.449518 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:56.449122    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:56.449518 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:56.449225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:56.727336 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.726378    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6c5gv" event={"ID":"584ee74d-2923-46fe-a902-cfffb43f7d32","Type":"ContainerStarted","Data":"bdf5cef85d4d77fcd81067f5d246ee33d5543e47ca172e21fa23d76400895b46"}
Sep 11 21:30:56.737150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.734660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qgbrm" event={"ID":"8b469ddc-2dbc-43b7-8719-8cc428baefef","Type":"ContainerStarted","Data":"434b9eeee16bc7ea3f5e618c2afb586e3e7e9038736f5f5f9fb6de44534d6207"}
Sep 11 21:30:56.741388 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.741224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bqqsd" event={"ID":"ac767800-1710-4e52-a859-2b3a9047369c","Type":"ContainerStarted","Data":"f6b956c6a7eb19551f6e6662974f9db0f5ec78f04c8b20dc96e604bb0432f085"}
Sep 11 21:30:56.743476 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.743251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6pfdg" event={"ID":"aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744","Type":"ContainerStarted","Data":"8e0ec575e365a27333722af7c4a8b6e7442afaff9a63f9bc0ff7d5c5ab11b24f"}
Sep 11 21:30:56.745429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.745347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"4bfa192fecb833c33fa911f32c3e2bca3f773b1fb6ab87487f82bfd13e981e8f"}
Sep 11 21:30:56.747958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.747906    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6c5gv" podStartSLOduration=2.768680663 podStartE2EDuration="21.74789251s" totalImagesPullingTime="18.979211847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:56.747373643 +0000 UTC m=+22.710398295" watchObservedRunningTime="2026-09-11 21:30:56.74789251 +0000 UTC m=+22.710917158"
Sep 11 21:30:56.748606 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.748429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-125.ec2.internal" podStartSLOduration=20.748417479 podStartE2EDuration="20.748417479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:39.71075996 +0000 UTC m=+5.673784614" watchObservedRunningTime="2026-09-11 21:30:56.748417479 +0000 UTC m=+22.711442128"
Sep 11 21:30:56.766359 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.766065    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6pfdg" podStartSLOduration=10.068431151 podStartE2EDuration="21.766049655s" totalImagesPullingTime="11.697618504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:56.766007119 +0000 UTC m=+22.729031769" watchObservedRunningTime="2026-09-11 21:30:56.766049655 +0000 UTC m=+22.729074300"
Sep 11 21:30:56.780782 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.780497    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bqqsd" podStartSLOduration=2.848936488 podStartE2EDuration="21.780481073s" totalImagesPullingTime="18.931544585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:56.780232301 +0000 UTC m=+22.743256959" watchObservedRunningTime="2026-09-11 21:30:56.780481073 +0000 UTC m=+22.743505722"
Sep 11 21:30:56.797333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:56.797162    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qgbrm" podStartSLOduration=2.818141854 podStartE2EDuration="21.797146974s" totalImagesPullingTime="18.97900512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:56.796836053 +0000 UTC m=+22.759860703" watchObservedRunningTime="2026-09-11 21:30:56.797146974 +0000 UTC m=+22.760171624"
Sep 11 21:30:57.448866 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.448787    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:57.449055 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:57.448929    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:57.750191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.750109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"fb7a1e4c2d61f605b36b6287836fec32f8aa45d5c6f15ca017cb4d03ac5bdeab"}
Sep 11 21:30:57.750191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.750185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"3dfa3fbd60407a448715ea782bd94d86a094923c346ab6b11a06350e17469365"}
Sep 11 21:30:57.750390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.750227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"355ce573b24edadc2ea4958fd755c6095c6550e3b0ab326317f4174e8abdc9fe"}
Sep 11 21:30:57.750390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.750257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"edf9ff26fee0d777820040cebf7904287d618d1b47322c7451378d63be96dd99"}
Sep 11 21:30:57.750390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.750282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"5b511f474fdb64896b287dc8e29e59a0684654832bee35b5d46e655d4c2c6c80"}
Sep 11 21:30:57.751612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.751590    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4xrb" event={"ID":"cc5971bb-454c-4913-b734-a524a351bb66","Type":"ContainerStarted","Data":"000b9bb760987319cb6f8e1160ec6d70025f79179fb1d2b66ff079e6da19d9f4"}
Sep 11 21:30:57.755891 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.755866    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="daf46de3755784a927615392cb5e9e588da642acbafce11e976637add359cb21" exitCode=0
Sep 11 21:30:57.756013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.755940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"daf46de3755784a927615392cb5e9e588da642acbafce11e976637add359cb21"}
Sep 11 21:30:57.759644 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.759611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw" event={"ID":"339eabf9-164e-4e88-b61e-f7ba222ec501","Type":"ContainerStarted","Data":"e844f9fc350414c93a44c2accf96df260cc857aef03658938c295015dcae583b"}
Sep 11 21:30:57.766961 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:57.766919    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h4xrb" podStartSLOduration=3.840934156 podStartE2EDuration="22.766904652s" totalImagesPullingTime="18.925970496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:57.766199685 +0000 UTC m=+23.729224333" watchObservedRunningTime="2026-09-11 21:30:57.766904652 +0000 UTC m=+23.729929302"
Sep 11 21:30:58.020819 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.020560    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:30:58.448828 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.448746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:30:58.448995 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.448746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:30:58.448995 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:58.448888    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:30:58.449114 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:58.448999    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:30:58.450902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.450810    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:30:58.020808338Z","UUID":"4e27649e-fbb2-44a4-9c81-c595a6c0613f","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:30:58.453413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.453392    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:30:58.453555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.453427    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:30:58.763566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.763486    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw" event={"ID":"339eabf9-164e-4e88-b61e-f7ba222ec501","Type":"ContainerStarted","Data":"8b7f32e2a1bf5f58da92501ee1d50353af30541fda05059437f4d378813aa22c"}
Sep 11 21:30:58.764902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.764880    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qjn7j" event={"ID":"202e2ea8-7802-492c-aa6d-24c814a08662","Type":"ContainerStarted","Data":"94548ff922053c3ed7d5e7a46a1af0d359d5f6bc41743d98d16ddb20d649c406"}
Sep 11 21:30:58.780659 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:58.780609    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qjn7j" podStartSLOduration=4.815758693 podStartE2EDuration="23.780590251s" totalImagesPullingTime="18.964831558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:58.779736163 +0000 UTC m=+24.742760815" watchObservedRunningTime="2026-09-11 21:30:58.780590251 +0000 UTC m=+24.743614917"
Sep 11 21:30:59.448955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.448882    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:30:59.449116 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:30:59.448985    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:30:59.770205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.770102    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw" event={"ID":"339eabf9-164e-4e88-b61e-f7ba222ec501","Type":"ContainerStarted","Data":"bd60289f44a8db82f9f676959af8dc58fad32f84ea0f6915a4110af08732f8a6"}
Sep 11 21:30:59.774065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.774030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"08fa4b592630f92342920af5438fcac206141f8cbe2609d6623b8955a7ab952d"}
Sep 11 21:30:59.792691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.792630    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wbfqw" podStartSLOduration=3.663374784 podStartE2EDuration="24.792610684s" totalImagesPullingTime="21.1292359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:30:59.792574079 +0000 UTC m=+25.755598729" watchObservedRunningTime="2026-09-11 21:30:59.792610684 +0000 UTC m=+25.755635327"
Sep 11 21:30:59.872125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.872086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:30:59.872821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:30:59.872794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:31:00.448691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:00.448647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:00.448884 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:00.448654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:00.448884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:00.448795    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:00.448884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:00.448871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:00.921301 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:00.921220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:31:00.921884 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:00.921864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6pfdg"
Sep 11 21:31:01.448348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:01.448313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:01.448513 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:01.448442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:31:02.448797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:02.448760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:02.449299 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:02.448760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:02.449299 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:02.448884    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:02.449299 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:02.449037    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:02.785228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:02.785030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" event={"ID":"b1a50d4a-fa78-4a02-8219-4dba86f42965","Type":"ContainerStarted","Data":"e4ae5f89da03804a86ffd6534347d642e7cfe94e745b3ae21087de917ac7905d"}
Sep 11 21:31:02.813690 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:02.813615    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl" podStartSLOduration=8.794491249 podStartE2EDuration="27.813599783s" totalImagesPullingTime="19.019108534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:02.813546919 +0000 UTC m=+28.776571569" watchObservedRunningTime="2026-09-11 21:31:02.813599783 +0000 UTC m=+28.776624431"
Sep 11 21:31:03.448627 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.448538    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:03.448833 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:03.448636    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:31:03.788544 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.788468    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="98d86f6fd71912075fa8c4dd017ee9794d4bed5924aec374674b92e6ab7cefb3" exitCode=0
Sep 11 21:31:03.788688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.788547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"98d86f6fd71912075fa8c4dd017ee9794d4bed5924aec374674b92e6ab7cefb3"}
Sep 11 21:31:03.789511 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.789146    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:03.789511 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.789206    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:03.789511 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.789250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:03.805328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.805302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:03.805461 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:03.805437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:04.449346 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:04.449314    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:04.449821 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:04.449427    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:04.449821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:04.449496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:04.449821 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:04.449612    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:05.094609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.094571    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w7md7"]
Sep 11 21:31:05.094799 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.094762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:05.094895 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:05.094872    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:05.097914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.097875    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shbrv"]
Sep 11 21:31:05.098097 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.097995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:05.098174 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:05.098091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:31:05.098550 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.098528    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7x2x"]
Sep 11 21:31:05.098643 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.098621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:05.098738 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:05.098702    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:05.795489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.795306    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="75f5d2073059e4d244d310107af8fe53b2b68402007dad5d2d6b9bf62ecb745a" exitCode=0
Sep 11 21:31:05.796010 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:05.795388    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"75f5d2073059e4d244d310107af8fe53b2b68402007dad5d2d6b9bf62ecb745a"}
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:06.448805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:06.448840    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:06.448868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:06.448957    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:06.449034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:31:06.449387 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:06.449111    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:06.799665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:06.799627    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="a851018c71db21d6472c8ad3235d16076bcbc5ca02a0d1628404b706ac904fad" exitCode=0
Sep 11 21:31:06.800110 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:06.799683    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"a851018c71db21d6472c8ad3235d16076bcbc5ca02a0d1628404b706ac904fad"}
Sep 11 21:31:08.253079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.253035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:08.253570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.253118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:08.253570 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.253205    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:31:08.253570 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.253228    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:31:08.253570 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.253293    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.253272159 +0000 UTC m=+66.216296805 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:31:08.253570 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.253341    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret podName:ee5ac432-0461-4ede-ac81-49fa12097464 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.253314052 +0000 UTC m=+66.216338701 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret") pod "global-pull-secret-syncer-shbrv" (UID: "ee5ac432-0461-4ede-ac81-49fa12097464") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:31:08.353971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.353935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:08.354173 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.354126    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:31:08.354173 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.354157    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:31:08.354173 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.354168    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fcg29 for pod openshift-network-diagnostics/network-check-target-w7md7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:31:08.354339 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.354228    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29 podName:64de28cd-ed89-42d0-9f1c-d2633f6844a3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:40.354214522 +0000 UTC m=+66.317239153 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcg29" (UniqueName: "kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29") pod "network-check-target-w7md7" (UID: "64de28cd-ed89-42d0-9f1c-d2633f6844a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:31:08.448979 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.448943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:08.449157 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.448942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:08.449157 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.449079    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-shbrv" podUID="ee5ac432-0461-4ede-ac81-49fa12097464"
Sep 11 21:31:08.449277 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.449161    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w7md7" podUID="64de28cd-ed89-42d0-9f1c-d2633f6844a3"
Sep 11 21:31:08.449277 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:08.448943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:08.449277 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:08.449268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7x2x" podUID="4dd4f841-9875-4dbd-be36-56e9a9bd205d"
Sep 11 21:31:09.332398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.332314    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-125.ec2.internal" event="NodeReady"
Sep 11 21:31:09.332805 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.332436    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:31:09.368783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.368756    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:31:09.370969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.370951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.373344 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.373319    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5txcp\""
Sep 11 21:31:09.373481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.373358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:31:09.373481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.373388    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:31:09.373481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.373319    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:31:09.379602 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.379115    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:31:09.386187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.386164    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:31:09.387856 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.387830    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r82z5"]
Sep 11 21:31:09.390076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.390059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.392227 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.392204    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:31:09.392345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.392241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:31:09.392345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.392211    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59g8b\""
Sep 11 21:31:09.393596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.393567    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mdgt2"]
Sep 11 21:31:09.395420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.395403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.398037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.397759    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vcb9z\""
Sep 11 21:31:09.398037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.397791    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:31:09.398037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.397809    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:31:09.398037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.397833    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:31:09.404011 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.403988    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r82z5"]
Sep 11 21:31:09.407100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.407046    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdgt2"]
Sep 11 21:31:09.463371 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463371 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-tmp-dir\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-config-volume\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-cert\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7c4k\" (UniqueName: \"kubernetes.io/projected/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-kube-api-access-j7c4k\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.463888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.463888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7lt\" (UniqueName: \"kubernetes.io/projected/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-kube-api-access-wd7lt\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.463888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.463772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmtm\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.564668 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.564604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.564866 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.564676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.564866 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.564791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.564866 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:09.564826    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:09.564866 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:09.564850    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:09.565066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.564921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-tmp-dir\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.565066 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:09.564936    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:10.06491392 +0000 UTC m=+36.027938551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:09.565066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.564982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-config-volume\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.565066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-cert\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.565066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7c4k\" (UniqueName: \"kubernetes.io/projected/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-kube-api-access-j7c4k\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7lt\" (UniqueName: \"kubernetes.io/projected/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-kube-api-access-wd7lt\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.565316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmtm\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565637 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.565911 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-config-volume\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.565911 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.565249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-tmp-dir\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.566066 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:09.565976    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:09.566066 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:09.566032    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:10.066015018 +0000 UTC m=+36.029039646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:09.566178 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.566105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.566412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.566377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.570613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.570586    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.570706 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.570594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.570706 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.570627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-cert\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.574240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.574191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.574506 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.574482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmtm\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:09.575079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.575054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7c4k\" (UniqueName: \"kubernetes.io/projected/5ac5b0a2-dc38-4cee-930c-0c06a85d8172-kube-api-access-j7c4k\") pod \"ingress-canary-mdgt2\" (UID: \"5ac5b0a2-dc38-4cee-930c-0c06a85d8172\") " pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:09.575502 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.575481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7lt\" (UniqueName: \"kubernetes.io/projected/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-kube-api-access-wd7lt\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:09.710767 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:09.710677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mdgt2"
Sep 11 21:31:10.069119 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.069075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:10.069302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.069138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:10.069302 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:10.069245    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:10.069302 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:10.069260    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:10.069302 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:10.069283    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:10.069302 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:10.069304    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:11.069288236 +0000 UTC m=+37.032312886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:10.069509 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:10.069345    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:11.069324537 +0000 UTC m=+37.032349170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:10.449089 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.449005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:10.449555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.449012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:10.449555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.449139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:10.451488 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.451463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:31:10.453177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.452284    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:31:10.453177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.452335    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:31:10.453177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.452494    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:31:10.453177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.452536    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r57v4\""
Sep 11 21:31:10.453177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:10.452288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxgvf\""
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.077073    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:11.077341    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:11.077360    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.077382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:11.077435    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:13.077408077 +0000 UTC m=+39.040432724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:11.077491    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:11.077573 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:11.077539    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:13.077524421 +0000 UTC m=+39.040549074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:11.295411 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.295315    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mdgt2"]
Sep 11 21:31:11.300239 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:11.300211    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac5b0a2_dc38_4cee_930c_0c06a85d8172.slice/crio-a2d7a7e0b1aba11d728381601107bfee8f9f89dfdd363ff3a23e83f7826947a6 WatchSource:0}: Error finding container a2d7a7e0b1aba11d728381601107bfee8f9f89dfdd363ff3a23e83f7826947a6: Status 404 returned error can't find the container with id a2d7a7e0b1aba11d728381601107bfee8f9f89dfdd363ff3a23e83f7826947a6
Sep 11 21:31:11.815454 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.815419    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="e9a53fa1a6c00e5dea09df16ea15db7ccd1b91ecf610275762f5a04e8704d077" exitCode=0
Sep 11 21:31:11.816006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.815477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"e9a53fa1a6c00e5dea09df16ea15db7ccd1b91ecf610275762f5a04e8704d077"}
Sep 11 21:31:11.816648 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:11.816627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdgt2" event={"ID":"5ac5b0a2-dc38-4cee-930c-0c06a85d8172","Type":"ContainerStarted","Data":"a2d7a7e0b1aba11d728381601107bfee8f9f89dfdd363ff3a23e83f7826947a6"}
Sep 11 21:31:12.104999 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:12.104897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:12.195797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:12.195760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:12.822575 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:12.822538    2582 generic.go:356] "Generic (PLEG): container finished" podID="b85435a6-55d3-4b79-997c-15a41ae02555" containerID="00ae5b5c9d7b8878469342050159e98f7de8f86ae491897cb2ce65d1e4b23d92" exitCode=0
Sep 11 21:31:12.823047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:12.822629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerDied","Data":"00ae5b5c9d7b8878469342050159e98f7de8f86ae491897cb2ce65d1e4b23d92"}
Sep 11 21:31:13.095123 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:13.095031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:13.095289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:13.095118    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:13.095289 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:13.095212    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:13.095392 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:13.095296    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:17.095272926 +0000 UTC m=+43.058297576 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:13.095392 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:13.095212    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:13.095392 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:13.095327    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:13.095392 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:13.095387    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:17.095373058 +0000 UTC m=+43.058397686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:13.828528 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:13.828497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gnknv" event={"ID":"b85435a6-55d3-4b79-997c-15a41ae02555","Type":"ContainerStarted","Data":"cd405984619c91036d591f631a7271b90ba4fd5153fd85733c6d222eaee47dbc"}
Sep 11 21:31:13.848810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:13.848763    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gnknv" podStartSLOduration=8.984970864 podStartE2EDuration="38.848748911s" totalImagesPullingTime="29.863778047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:13.847389798 +0000 UTC m=+39.810414447" watchObservedRunningTime="2026-09-11 21:31:13.848748911 +0000 UTC m=+39.811773583"
Sep 11 21:31:13.968737 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:13.968691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.053459 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.053429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.116214 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.116134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.171216 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.171182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.748782 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.748544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.817914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.817800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:14.832933 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.832895    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mdgt2" event={"ID":"5ac5b0a2-dc38-4cee-930c-0c06a85d8172","Type":"ContainerStarted","Data":"a63ba2b41546366ea7591a7f3676b760b3958f2ce6754d62ee4a30c292c5b0a1"}
Sep 11 21:31:14.849817 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:14.849763    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mdgt2" podStartSLOduration=2.977711028 podStartE2EDuration="5.849747559s" totalImagesPullingTime="2.872036531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:14.848983729 +0000 UTC m=+40.812008378" watchObservedRunningTime="2026-09-11 21:31:14.849747559 +0000 UTC m=+40.812772218"
Sep 11 21:31:16.689907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:16.689868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:16.790749 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:16.790697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:17.123828 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:17.123797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:17.124006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:17.123854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:17.124006 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:17.123967    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:17.124006 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:17.123997    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:17.124006 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:17.123971    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:17.124137 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:17.124057    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:25.124042161 +0000 UTC m=+51.087066791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:17.124137 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:17.124070    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:25.12406461 +0000 UTC m=+51.087089237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:24.198077 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:24.198039    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:24.260187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:24.260151    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:24.323102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:24.323074    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:25.072067 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.072033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:25.170500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.170461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:25.184002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.183976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:25.184196 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.184041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:25.184196 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:25.184135    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:25.184312 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:25.184197    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:31:41.184177271 +0000 UTC m=+67.147201911 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:25.184312 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:25.184136    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:25.184312 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:25.184235    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:25.184312 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:25.184285    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:31:41.184273219 +0000 UTC m=+67.147297867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:25.393255 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.393168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:25.464433 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:25.464387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:35.184195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:35.184160    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:35.806377 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:35.806354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trnpl"
Sep 11 21:31:35.961161 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:35.961099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:36.063031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.062930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:36.955026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.954970    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:36.985068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.985035    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"]
Sep 11 21:31:36.987737 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.987702    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"]
Sep 11 21:31:36.987852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.987837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:36.989347 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.989324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"
Sep 11 21:31:36.990932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.990913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pxswf\""
Sep 11 21:31:36.991022 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.990948    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 11 21:31:36.991398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.991380    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 11 21:31:36.991569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.991548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 11 21:31:36.991732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.991622    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:31:36.992091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.992054    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vn25h\""
Sep 11 21:31:36.999569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:36.999533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"]
Sep 11 21:31:37.000500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.000391    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"]
Sep 11 21:31:37.039083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.039048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:37.068088 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.068054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpm4g\" (UniqueName: \"kubernetes.io/projected/86072044-4771-4739-9eb1-27a0963b134e-kube-api-access-vpm4g\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.068088 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.068092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrs8\" (UniqueName: \"kubernetes.io/projected/1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c-kube-api-access-hmrs8\") pod \"network-check-source-6f8b768455-pf6th\" (UID: \"1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"
Sep 11 21:31:37.068300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.068170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86072044-4771-4739-9eb1-27a0963b134e-config\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.068300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.068235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86072044-4771-4739-9eb1-27a0963b134e-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.085554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.085532    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-wv46w"]
Sep 11 21:31:37.087390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.087373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.092548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.092523    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 21:31:37.092988 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.092965    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:31:37.094358 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.094338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 21:31:37.095301 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.095284    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:31:37.099259 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.099241    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xp4lg\""
Sep 11 21:31:37.109948 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.109927    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wv46w"]
Sep 11 21:31:37.111325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.111306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 21:31:37.168541 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86072044-4771-4739-9eb1-27a0963b134e-config\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-tmp\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86072044-4771-4739-9eb1-27a0963b134e-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpm4g\" (UniqueName: \"kubernetes.io/projected/86072044-4771-4739-9eb1-27a0963b134e-kube-api-access-vpm4g\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrs8\" (UniqueName: \"kubernetes.io/projected/1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c-kube-api-access-hmrs8\") pod \"network-check-source-6f8b768455-pf6th\" (UID: \"1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"
Sep 11 21:31:37.168766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-service-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.169023 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24d43615-b6a3-40ff-a376-4c095814d988-serving-cert\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.169023 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-snapshots\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.169023 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.168836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zxs\" (UniqueName: \"kubernetes.io/projected/24d43615-b6a3-40ff-a376-4c095814d988-kube-api-access-z5zxs\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.169223 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.169202    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/86072044-4771-4739-9eb1-27a0963b134e-config\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.172189 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.172171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/86072044-4771-4739-9eb1-27a0963b134e-serving-cert\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.183058 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.183030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpm4g\" (UniqueName: \"kubernetes.io/projected/86072044-4771-4739-9eb1-27a0963b134e-kube-api-access-vpm4g\") pod \"service-ca-operator-cf7c776b7-xnhpj\" (UID: \"86072044-4771-4739-9eb1-27a0963b134e\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.185117 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.185103    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:31:37.196203 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.196180    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:31:37.206583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.206536    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrs8\" (UniqueName: \"kubernetes.io/projected/1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c-kube-api-access-hmrs8\") pod \"network-check-source-6f8b768455-pf6th\" (UID: \"1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"
Sep 11 21:31:37.269225 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-service-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269225 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24d43615-b6a3-40ff-a376-4c095814d988-serving-cert\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-snapshots\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zxs\" (UniqueName: \"kubernetes.io/projected/24d43615-b6a3-40ff-a376-4c095814d988-kube-api-access-z5zxs\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269387    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-tmp\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.269929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.269901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-tmp\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.270058 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.270003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24d43615-b6a3-40ff-a376-4c095814d988-snapshots\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.270058 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.270009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-service-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.270230 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.270209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24d43615-b6a3-40ff-a376-4c095814d988-trusted-ca-bundle\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.271661 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.271646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24d43615-b6a3-40ff-a376-4c095814d988-serving-cert\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.280832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.280810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zxs\" (UniqueName: \"kubernetes.io/projected/24d43615-b6a3-40ff-a376-4c095814d988-kube-api-access-z5zxs\") pod \"insights-operator-5844b856db-wv46w\" (UID: \"24d43615-b6a3-40ff-a376-4c095814d988\") " pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.297763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.297743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"
Sep 11 21:31:37.303242 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.303208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"
Sep 11 21:31:37.360560 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.360521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:37.396093 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.396057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-wv46w"
Sep 11 21:31:37.455645 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.455609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:37.572098 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.572069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj"]
Sep 11 21:31:37.575076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.574975    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pf6th"]
Sep 11 21:31:37.575267 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:37.575243    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86072044_4771_4739_9eb1_27a0963b134e.slice/crio-debc226602676ae4ae3b130cc2eda833e7f2e78464ecf081d4bb56f4bf7bf6cc WatchSource:0}: Error finding container debc226602676ae4ae3b130cc2eda833e7f2e78464ecf081d4bb56f4bf7bf6cc: Status 404 returned error can't find the container with id debc226602676ae4ae3b130cc2eda833e7f2e78464ecf081d4bb56f4bf7bf6cc
Sep 11 21:31:37.577346 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:37.577326    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbb0807_8d1e_4503_a96a_b3e1cc38cf3c.slice/crio-e9227fa8b3a351c269707cc394d7ec839e112cba6d3ab5872534777f085d570e WatchSource:0}: Error finding container e9227fa8b3a351c269707cc394d7ec839e112cba6d3ab5872534777f085d570e: Status 404 returned error can't find the container with id e9227fa8b3a351c269707cc394d7ec839e112cba6d3ab5872534777f085d570e
Sep 11 21:31:37.673975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.673946    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-wv46w"]
Sep 11 21:31:37.677343 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:37.677315    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d43615_b6a3_40ff_a376_4c095814d988.slice/crio-f79dd87c5835bc397d26a7e7f4914f4e59e6dd1f746f56c6507701bb53817369 WatchSource:0}: Error finding container f79dd87c5835bc397d26a7e7f4914f4e59e6dd1f746f56c6507701bb53817369: Status 404 returned error can't find the container with id f79dd87c5835bc397d26a7e7f4914f4e59e6dd1f746f56c6507701bb53817369
Sep 11 21:31:37.889238 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.889143    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj" event={"ID":"86072044-4771-4739-9eb1-27a0963b134e","Type":"ContainerStarted","Data":"debc226602676ae4ae3b130cc2eda833e7f2e78464ecf081d4bb56f4bf7bf6cc"}
Sep 11 21:31:37.890326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.890300    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wv46w" event={"ID":"24d43615-b6a3-40ff-a376-4c095814d988","Type":"ContainerStarted","Data":"f79dd87c5835bc397d26a7e7f4914f4e59e6dd1f746f56c6507701bb53817369"}
Sep 11 21:31:37.891332 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:37.891313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th" event={"ID":"1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c","Type":"ContainerStarted","Data":"e9227fa8b3a351c269707cc394d7ec839e112cba6d3ab5872534777f085d570e"}
Sep 11 21:31:39.215887 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:39.215850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:39.278860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:39.278826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:40.294212 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.294179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:31:40.294640 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.294265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:40.296904 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.296884    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:31:40.296952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.296891    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:31:40.304667 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:40.304642    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 21:31:40.304822 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:40.304730    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs podName:4dd4f841-9875-4dbd-be36-56e9a9bd205d nodeName:}" failed. No retries permitted until 2026-09-11 21:32:44.304690961 +0000 UTC m=+130.267715602 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs") pod "network-metrics-daemon-x7x2x" (UID: "4dd4f841-9875-4dbd-be36-56e9a9bd205d") : secret "metrics-daemon-secret" not found
Sep 11 21:31:40.307015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.306988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ee5ac432-0461-4ede-ac81-49fa12097464-original-pull-secret\") pod \"global-pull-secret-syncer-shbrv\" (UID: \"ee5ac432-0461-4ede-ac81-49fa12097464\") " pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:40.395623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.395582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:40.398602 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.398575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcg29\" (UniqueName: \"kubernetes.io/projected/64de28cd-ed89-42d0-9f1c-d2633f6844a3-kube-api-access-fcg29\") pod \"network-check-target-w7md7\" (UID: \"64de28cd-ed89-42d0-9f1c-d2633f6844a3\") " pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:40.464026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.463989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wxgvf\""
Sep 11 21:31:40.468908 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.468884    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-shbrv"
Sep 11 21:31:40.471953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:40.471931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:41.200646 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.200601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:31:41.200895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.200668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:31:41.200895 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:41.200832    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:31:41.200895 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:41.200836    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 21:31:41.200895 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:41.200863    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69dc887c94-68psq: secret "image-registry-tls" not found
Sep 11 21:31:41.200895 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:41.200895    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls podName:e1f27b02-e4f3-41ad-9efe-5c24fb5a904f nodeName:}" failed. No retries permitted until 2026-09-11 21:32:13.200876365 +0000 UTC m=+99.163901032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls") pod "dns-default-r82z5" (UID: "e1f27b02-e4f3-41ad-9efe-5c24fb5a904f") : secret "dns-default-metrics-tls" not found
Sep 11 21:31:41.201148 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:41.200922    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls podName:03dd5e85-1079-460e-acf5-d5df9bac46bc nodeName:}" failed. No retries permitted until 2026-09-11 21:32:13.200902803 +0000 UTC m=+99.163927447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls") pod "image-registry-69dc887c94-68psq" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc") : secret "image-registry-tls" not found
Sep 11 21:31:41.378906 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.378846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:41.475237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.475143    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:41.555362 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.555330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-shbrv"]
Sep 11 21:31:41.559297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.559267    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w7md7"]
Sep 11 21:31:41.562100 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:41.561981    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5ac432_0461_4ede_ac81_49fa12097464.slice/crio-f38e6c769d13b2177de0fff04acbd86ef228b8634c9d7276552142675901593b WatchSource:0}: Error finding container f38e6c769d13b2177de0fff04acbd86ef228b8634c9d7276552142675901593b: Status 404 returned error can't find the container with id f38e6c769d13b2177de0fff04acbd86ef228b8634c9d7276552142675901593b
Sep 11 21:31:41.564390 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:31:41.564369    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64de28cd_ed89_42d0_9f1c_d2633f6844a3.slice/crio-6449f7ca538326ed4e48aebb7294d377a5318b2e8f465beb85d4fce3e618a031 WatchSource:0}: Error finding container 6449f7ca538326ed4e48aebb7294d377a5318b2e8f465beb85d4fce3e618a031: Status 404 returned error can't find the container with id 6449f7ca538326ed4e48aebb7294d377a5318b2e8f465beb85d4fce3e618a031
Sep 11 21:31:41.905372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.905337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj" event={"ID":"86072044-4771-4739-9eb1-27a0963b134e","Type":"ContainerStarted","Data":"1e7f96600c095934ac3ea1cfd7794e3932ab345d62c3112b00ddda02e73717d5"}
Sep 11 21:31:41.906911 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.906861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shbrv" event={"ID":"ee5ac432-0461-4ede-ac81-49fa12097464","Type":"ContainerStarted","Data":"f38e6c769d13b2177de0fff04acbd86ef228b8634c9d7276552142675901593b"}
Sep 11 21:31:41.908819 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.908795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wv46w" event={"ID":"24d43615-b6a3-40ff-a376-4c095814d988","Type":"ContainerStarted","Data":"585110c616c4ab9f355e899c39c35e05fb45fe2b43ea3038c7bbce12308c8c35"}
Sep 11 21:31:41.910764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.910706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th" event={"ID":"1dbb0807-8d1e-4503-a96a-b3e1cc38cf3c","Type":"ContainerStarted","Data":"2815afef0d91a25cdf850342ebd07572cb440dcbf3fc5a259ed3a72bcc0048b3"}
Sep 11 21:31:41.912373 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.912349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w7md7" event={"ID":"64de28cd-ed89-42d0-9f1c-d2633f6844a3","Type":"ContainerStarted","Data":"6fa6fabcdcec010b2624bf55398e60793c1d9e5f8e5808c07ea6814cb7a9e7c5"}
Sep 11 21:31:41.912470 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.912389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w7md7" event={"ID":"64de28cd-ed89-42d0-9f1c-d2633f6844a3","Type":"ContainerStarted","Data":"6449f7ca538326ed4e48aebb7294d377a5318b2e8f465beb85d4fce3e618a031"}
Sep 11 21:31:41.912470 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.912459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:31:41.921210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.921152    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj" podStartSLOduration=2.212762896 podStartE2EDuration="5.9211343s" totalImagesPullingTime="3.708371404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:41.920951219 +0000 UTC m=+67.883975869" watchObservedRunningTime="2026-09-11 21:31:41.9211343 +0000 UTC m=+67.884158950"
Sep 11 21:31:41.936176 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.936121    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pf6th" podStartSLOduration=2.224072252 podStartE2EDuration="5.936100743s" totalImagesPullingTime="3.712028491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:41.93589199 +0000 UTC m=+67.898916639" watchObservedRunningTime="2026-09-11 21:31:41.936100743 +0000 UTC m=+67.899125393"
Sep 11 21:31:41.951574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.951523    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-wv46w" podStartSLOduration=1.343691114 podStartE2EDuration="4.95150595s" totalImagesPullingTime="3.607814836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:41.950743891 +0000 UTC m=+67.913768541" watchObservedRunningTime="2026-09-11 21:31:41.95150595 +0000 UTC m=+67.914530986"
Sep 11 21:31:41.965419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:41.965368    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w7md7" podStartSLOduration=66.965351617 podStartE2EDuration="1m6.965351617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:41.964327872 +0000 UTC m=+67.927352534" watchObservedRunningTime="2026-09-11 21:31:41.965351617 +0000 UTC m=+67.928376266"
Sep 11 21:31:42.404187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.404141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:42.472569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.472533    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dqb5r"]
Sep 11 21:31:42.474914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.474890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.478068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.478043    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:31:42.478200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.478149    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hpqdp\""
Sep 11 21:31:42.482613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.482591    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:31:42.496617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.496591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqb5r"]
Sep 11 21:31:42.512633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.512605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llq68\" (UniqueName: \"kubernetes.io/projected/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-api-access-llq68\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.512912 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.512890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.513051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.513029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.513146 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.513087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-crio-socket\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.513146 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.513117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-data-volume\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.521831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.521804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:42.587130 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.587094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:42.614160 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llq68\" (UniqueName: \"kubernetes.io/projected/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-api-access-llq68\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614160 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-crio-socket\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-data-volume\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614398 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:42.614303    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:42.614398 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:42.614374    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.114352255 +0000 UTC m=+69.077376898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:42.614649 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:42.614556    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:42.614649 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:42.614609    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:43.114593775 +0000 UTC m=+69.077618422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:42.614649 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-data-volume\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.614836 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.614673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-crio-socket\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.623922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.623898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llq68\" (UniqueName: \"kubernetes.io/projected/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-api-access-llq68\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:42.917229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:42.917194    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:43.018608 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.018578    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:43.118613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.118581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:43.118613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.118617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:43.118884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:43.118709    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:43.118884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:43.118776    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:43.118884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:43.118793    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.118777332 +0000 UTC m=+70.081801960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:43.118884 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:43.118833    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:44.11881922 +0000 UTC m=+70.081843862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:43.354054 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.354017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:43.414253 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.414214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:43.638567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.638469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:43.738663 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:43.738623    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:44.126545 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:44.126512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:44.126751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:44.126553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:44.126751 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:44.126654    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:44.126751 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:44.126689    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:44.126751 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:44.126731    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:46.126700821 +0000 UTC m=+72.089725461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:44.126927 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:44.126782    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:46.126762053 +0000 UTC m=+72.089786692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:45.238110 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:45.238071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:45.329202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:45.329163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:45.334442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:45.334418    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4xrb_cc5971bb-454c-4913-b734-a524a351bb66/dns-node-resolver/0.log"
Sep 11 21:31:46.143988 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.143957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:46.143988 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.143992    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:46.144185 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:46.144119    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:46.144185 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:46.144159    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:46.144249 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:46.144186    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:50.144169789 +0000 UTC m=+76.107194433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:46.144249 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:46.144212    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:50.144198086 +0000 UTC m=+76.107222728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:46.182404 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.182367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:46.335316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.335253    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bqqsd_ac767800-1710-4e52-a859-2b3a9047369c/node-ca/0.log"
Sep 11 21:31:46.929582 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.929503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-shbrv" event={"ID":"ee5ac432-0461-4ede-ac81-49fa12097464","Type":"ContainerStarted","Data":"fa8591779a6dfb54f6e3b1c968bfeefeb4d95fac0e217e14418ff5895f569c8a"}
Sep 11 21:31:46.945257 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:46.945203    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-shbrv" podStartSLOduration=66.935505349 podStartE2EDuration="1m11.945188515s" totalImagesPullingTime="5.009683166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:31:46.944303518 +0000 UTC m=+72.907328167" watchObservedRunningTime="2026-09-11 21:31:46.945188515 +0000 UTC m=+72.908213204"
Sep 11 21:31:47.335480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:47.335446    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mdgt2_5ac5b0a2-dc38-4cee-930c-0c06a85d8172/serve-healthcheck-canary/0.log"
Sep 11 21:31:49.571777 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:49.571741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:49.641300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:49.641262    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:49.718659 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:49.718614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:50.176224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:50.176185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:50.176224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:50.176225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:50.176460 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:50.176322    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:31:50.176460 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:50.176332    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:50.176460 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:50.176391    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:58.176371264 +0000 UTC m=+84.139395892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : configmap "kube-rbac-proxy" not found
Sep 11 21:31:50.176460 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:50.176409    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:31:58.176399587 +0000 UTC m=+84.139424215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:51.463845 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:51.463809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:51.563766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:51.563734    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:51.837587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:51.837550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:51.929163 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:51.929130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:52.154821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:52.154728    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:52.633227 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:52.633179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:53.356115 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:53.356080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:53.558675 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:53.558642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:53.759467 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:53.759411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:53.971272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:53.971238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:54.158101 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:54.157993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:54.521060 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:54.521024    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:54.964232 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:54.964141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.072730 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.072687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.174658 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.174601    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.275159 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.275120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.372791 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.372757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.685577 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.685400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.786365 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.786331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:55.961322 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:55.961234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:56.159129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:56.159090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:56.220665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:56.220582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:57.161889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:57.161845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:57.566109 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:57.566072    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:57.763609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:57.763575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:57.958740 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:57.958630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:58.072197 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.072163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:58.240391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.240365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:58.240793 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.240400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:58.240793 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:58.240523    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:58.240793 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:31:58.240592    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls podName:5fc0e383-1643-48e2-ac95-ecb52dd72ac3 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:14.240575246 +0000 UTC m=+100.203599873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dqb5r" (UID: "5fc0e383-1643-48e2-ac95-ecb52dd72ac3") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:31:58.240974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.240957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:31:58.432583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.432544    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:58.957250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:58.957217    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:59.559617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:59.559577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:59.764230 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:59.764199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:31:59.971438 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:31:59.971344    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:00.361499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:00.361474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:00.758200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:00.758170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:01.363572 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:01.363536    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:01.764985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:01.764952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:01.961681 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:01.961646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:02.163298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:02.163202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.212003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.211953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.316116 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.316012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.493813 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.493775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.575005 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.574968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.700511 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.700477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.754940 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.754864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.890900 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.890865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:03.991587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:03.991554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:04.922028 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:04.921995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:04.980447 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:04.980414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:05.101157 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:05.101128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:05.317909 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:05.317849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:08.979777 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.979738    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"]
Sep 11 21:32:08.982348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.982328    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:08.984654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.984631    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:32:08.985391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.985364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:32:08.985391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.985386    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 21:32:08.985569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.985371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:32:08.985569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:08.985371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-f92qk\""
Sep 11 21:32:09.002429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.002403    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"]
Sep 11 21:32:09.111594 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.111563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:09.126774 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.126742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.126916 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.126815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxfp\" (UniqueName: \"kubernetes.io/projected/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-kube-api-access-5lxfp\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.207444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.207410    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:09.227802 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.227769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxfp\" (UniqueName: \"kubernetes.io/projected/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-kube-api-access-5lxfp\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.227972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.227884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.230660 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.230602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.245781 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.245753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxfp\" (UniqueName: \"kubernetes.io/projected/b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6-kube-api-access-5lxfp\") pod \"managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl\" (UID: \"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.299782 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.299747    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"
Sep 11 21:32:09.570380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:09.570311    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl"]
Sep 11 21:32:09.573745 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:09.573691    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb05c98aa_a8d1_4959_b4b2_5da1d2d52ba6.slice/crio-c7cc35fdddc20917ac7535ab3d2a0fe7ed4a1dd9e3943004af8be27029174f3f WatchSource:0}: Error finding container c7cc35fdddc20917ac7535ab3d2a0fe7ed4a1dd9e3943004af8be27029174f3f: Status 404 returned error can't find the container with id c7cc35fdddc20917ac7535ab3d2a0fe7ed4a1dd9e3943004af8be27029174f3f
Sep 11 21:32:10.000458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.000415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl" event={"ID":"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6","Type":"ContainerStarted","Data":"c7cc35fdddc20917ac7535ab3d2a0fe7ed4a1dd9e3943004af8be27029174f3f"}
Sep 11 21:32:10.741641 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.741603    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:10.743736 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.743694    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.746001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.745982    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 21:32:10.746818 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746796    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 21:32:10.746913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746836    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 21:32:10.746913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746862    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jg727\""
Sep 11 21:32:10.747034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746836    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 21:32:10.747034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746795    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 21:32:10.747034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746842    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 21:32:10.747034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.746836    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 21:32:10.760811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.760790    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:10.840282 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.840463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.840463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.840463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.840624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.840624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.840506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpjd\" (UniqueName: \"kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.940946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.940912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.940946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.940952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.941200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.940979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.941200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.941200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.941200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpjd\" (UniqueName: \"kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.941875 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.942002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.942002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.941916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.943901 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.943878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.944299 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.944277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:10.949465 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:10.949422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpjd\" (UniqueName: \"kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd\") pod \"console-f8c48665c-x5h7m\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") " pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:11.055940 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:11.055863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:11.327974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:11.327892    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:11.888106 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:11.888072    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57ac5dd_28ab_4e7a_9997_3d9c03686932.slice/crio-21cf6a979232eade610ea5ff76e6d13ad720e2d775c428db7797f20e313a6e9e WatchSource:0}: Error finding container 21cf6a979232eade610ea5ff76e6d13ad720e2d775c428db7797f20e313a6e9e: Status 404 returned error can't find the container with id 21cf6a979232eade610ea5ff76e6d13ad720e2d775c428db7797f20e313a6e9e
Sep 11 21:32:12.010417 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:12.010377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c48665c-x5h7m" event={"ID":"f57ac5dd-28ab-4e7a-9997-3d9c03686932","Type":"ContainerStarted","Data":"21cf6a979232eade610ea5ff76e6d13ad720e2d775c428db7797f20e313a6e9e"}
Sep 11 21:32:12.735468 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:12.735425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:12.793859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:12.793817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:12.918266 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:12.918236    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w7md7"
Sep 11 21:32:13.015823 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.015744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl" event={"ID":"b05c98aa-a8d1-4959-b4b2-5da1d2d52ba6","Type":"ContainerStarted","Data":"fd7d7f98d922af6da4aa4452cef579744314ecea5cce9472c0fe2edd0d3545bf"}
Sep 11 21:32:13.031324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.031270    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5ffcfbb48d-95lpl" podStartSLOduration=2.354375538 podStartE2EDuration="5.031250797s" totalImagesPullingTime="2.676875259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:13.030399387 +0000 UTC m=+98.993424061" watchObservedRunningTime="2026-09-11 21:32:13.031250797 +0000 UTC m=+98.994275445"
Sep 11 21:32:13.260649 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.260615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:13.260870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.260732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:32:13.263479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.263435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1f27b02-e4f3-41ad-9efe-5c24fb5a904f-metrics-tls\") pod \"dns-default-r82z5\" (UID: \"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f\") " pod="openshift-dns/dns-default-r82z5"
Sep 11 21:32:13.263609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.263556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"image-registry-69dc887c94-68psq\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") " pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:13.284531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.284468    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5txcp\""
Sep 11 21:32:13.293493 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.293473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:13.305862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.305835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-59g8b\""
Sep 11 21:32:13.313756 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.313709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r82z5"
Sep 11 21:32:13.596151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.596045    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:32:13.599941 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:13.599875    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dd5e85_1079_460e_acf5_d5df9bac46bc.slice/crio-6bebfce850b58b7718ffc263d73626a7a975393e8669a4551b861549751413f5 WatchSource:0}: Error finding container 6bebfce850b58b7718ffc263d73626a7a975393e8669a4551b861549751413f5: Status 404 returned error can't find the container with id 6bebfce850b58b7718ffc263d73626a7a975393e8669a4551b861549751413f5
Sep 11 21:32:13.620617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.620586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r82z5"]
Sep 11 21:32:13.623911 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:13.623883    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f27b02_e4f3_41ad_9efe_5c24fb5a904f.slice/crio-03cc15ac8c32aeb70c5d16d640cd84404cc2bdf61f17dce91714d9f3218d6a19 WatchSource:0}: Error finding container 03cc15ac8c32aeb70c5d16d640cd84404cc2bdf61f17dce91714d9f3218d6a19: Status 404 returned error can't find the container with id 03cc15ac8c32aeb70c5d16d640cd84404cc2bdf61f17dce91714d9f3218d6a19
Sep 11 21:32:13.700378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.700328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:13.803353 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.803315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:13.856474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.856308    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"]
Sep 11 21:32:13.861941 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.861914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:13.863936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.863912    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 21:32:13.864079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.864002    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nt7zj\""
Sep 11 21:32:13.866635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.866612    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"]
Sep 11 21:32:13.873418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.873395    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:13.967479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:13.967440    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/558736c1-1f2d-4323-aed2-78f43ea8f769-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jngs\" (UID: \"558736c1-1f2d-4323-aed2-78f43ea8f769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:14.021189 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.021154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r82z5" event={"ID":"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f","Type":"ContainerStarted","Data":"03cc15ac8c32aeb70c5d16d640cd84404cc2bdf61f17dce91714d9f3218d6a19"}
Sep 11 21:32:14.023752 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.023696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dc887c94-68psq" event={"ID":"03dd5e85-1079-460e-acf5-d5df9bac46bc","Type":"ContainerStarted","Data":"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"}
Sep 11 21:32:14.023905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.023772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dc887c94-68psq" event={"ID":"03dd5e85-1079-460e-acf5-d5df9bac46bc","Type":"ContainerStarted","Data":"6bebfce850b58b7718ffc263d73626a7a975393e8669a4551b861549751413f5"}
Sep 11 21:32:14.037779 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.037752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:14.044751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.044691    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69dc887c94-68psq" podStartSLOduration=99.044676018 podStartE2EDuration="1m39.044676018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:14.043888761 +0000 UTC m=+100.006913424" watchObservedRunningTime="2026-09-11 21:32:14.044676018 +0000 UTC m=+100.007700667"
Sep 11 21:32:14.068699 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.068654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/558736c1-1f2d-4323-aed2-78f43ea8f769-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jngs\" (UID: \"558736c1-1f2d-4323-aed2-78f43ea8f769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:14.071557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.071535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/558736c1-1f2d-4323-aed2-78f43ea8f769-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jngs\" (UID: \"558736c1-1f2d-4323-aed2-78f43ea8f769\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:14.151338 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.151240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:14.174024 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.173998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:14.214807 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.214309    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:14.272306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.271019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:32:14.274905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.274848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fc0e383-1643-48e2-ac95-ecb52dd72ac3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dqb5r\" (UID: \"5fc0e383-1643-48e2-ac95-ecb52dd72ac3\") " pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:32:14.284693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:14.284664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dqb5r"
Sep 11 21:32:15.027485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:15.027456    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:16.563376 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:16.563347    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"]
Sep 11 21:32:16.566111 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:16.566084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558736c1_1f2d_4323_aed2_78f43ea8f769.slice/crio-30fd3e377accb9b79964d13aa107ab630406a7830f44ad798a0e6300aca52c6d WatchSource:0}: Error finding container 30fd3e377accb9b79964d13aa107ab630406a7830f44ad798a0e6300aca52c6d: Status 404 returned error can't find the container with id 30fd3e377accb9b79964d13aa107ab630406a7830f44ad798a0e6300aca52c6d
Sep 11 21:32:16.566309 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:16.566289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dqb5r"]
Sep 11 21:32:16.569558 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:16.569536    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc0e383_1643_48e2_ac95_ecb52dd72ac3.slice/crio-0d7c95a502dc3a3d5d4696762c002a5827787bf0816c701f05eac8fa10802bb6 WatchSource:0}: Error finding container 0d7c95a502dc3a3d5d4696762c002a5827787bf0816c701f05eac8fa10802bb6: Status 404 returned error can't find the container with id 0d7c95a502dc3a3d5d4696762c002a5827787bf0816c701f05eac8fa10802bb6
Sep 11 21:32:16.925454 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:16.925355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:16.980068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:16.980032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:17.036220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.036180    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c48665c-x5h7m" event={"ID":"f57ac5dd-28ab-4e7a-9997-3d9c03686932","Type":"ContainerStarted","Data":"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"}
Sep 11 21:32:17.037835 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.037813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqb5r" event={"ID":"5fc0e383-1643-48e2-ac95-ecb52dd72ac3","Type":"ContainerStarted","Data":"35be532a9169a47d4e259132339d54f6bf11891e53b4444ef83ee3249da46720"}
Sep 11 21:32:17.037899 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.037872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqb5r" event={"ID":"5fc0e383-1643-48e2-ac95-ecb52dd72ac3","Type":"ContainerStarted","Data":"0d7c95a502dc3a3d5d4696762c002a5827787bf0816c701f05eac8fa10802bb6"}
Sep 11 21:32:17.039055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.039035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs" event={"ID":"558736c1-1f2d-4323-aed2-78f43ea8f769","Type":"ContainerStarted","Data":"30fd3e377accb9b79964d13aa107ab630406a7830f44ad798a0e6300aca52c6d"}
Sep 11 21:32:17.040746 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.040708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r82z5" event={"ID":"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f","Type":"ContainerStarted","Data":"1e4d9cc65b6ca312a1e4e2544260d86cd055fe72e7109c3f39ce44d86228715c"}
Sep 11 21:32:17.040889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.040766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r82z5" event={"ID":"e1f27b02-e4f3-41ad-9efe-5c24fb5a904f","Type":"ContainerStarted","Data":"32a84c49abcee72ef5e7fb55c88b31770b7f2ff1909432172658a809950a2e6a"}
Sep 11 21:32:17.040889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.040872    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r82z5"
Sep 11 21:32:17.055286 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.055238    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f8c48665c-x5h7m" podStartSLOduration=2.654461898 podStartE2EDuration="7.055223776s" totalImagesPullingTime="4.400761878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:17.054127341 +0000 UTC m=+103.017152015" watchObservedRunningTime="2026-09-11 21:32:17.055223776 +0000 UTC m=+103.018248430"
Sep 11 21:32:17.076328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:17.076277    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r82z5" podStartSLOduration=65.404849976 podStartE2EDuration="1m8.076258251s" totalImagesPullingTime="2.671408275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:17.075481295 +0000 UTC m=+103.038505945" watchObservedRunningTime="2026-09-11 21:32:17.076258251 +0000 UTC m=+103.039282899"
Sep 11 21:32:19.050609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.050569    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqb5r" event={"ID":"5fc0e383-1643-48e2-ac95-ecb52dd72ac3","Type":"ContainerStarted","Data":"fbec2df1dda3f769ee8d2e63a4b581abdd49a332cc0da70fa846d5ffd93d0a2b"}
Sep 11 21:32:19.052253 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.052219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs" event={"ID":"558736c1-1f2d-4323-aed2-78f43ea8f769","Type":"ContainerStarted","Data":"6db77b01b1c271e6b65ae2f20dff9892d20f86fba304f9cb5ebff8865af1e40b"}
Sep 11 21:32:19.052424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.052406    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:19.057619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.057596    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs"
Sep 11 21:32:19.069686 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.069634    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jngs" podStartSLOduration=4.273952187 podStartE2EDuration="6.06961354s" totalImagesPullingTime="1.795661353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:19.069019528 +0000 UTC m=+105.032044183" watchObservedRunningTime="2026-09-11 21:32:19.06961354 +0000 UTC m=+105.032638207"
Sep 11 21:32:19.731692 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.731666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:19.794861 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.794823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:19.897779 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.897744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:19.922660 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.922625    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"]
Sep 11 21:32:19.924742 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.924704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:19.928361 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928336    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 21:32:19.928676 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928372    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:32:19.928676 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928337    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 21:32:19.928676 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:32:19.928894 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928746    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:32:19.928976 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.928962    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sqg8p\""
Sep 11 21:32:19.935708 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.935685    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"]
Sep 11 21:32:19.971931 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:19.971895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:20.018049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.017975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.018049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.018010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jblc\" (UniqueName: \"kubernetes.io/projected/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-kube-api-access-2jblc\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.018217 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.018093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.018217 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.018146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.057289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.057243    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dqb5r" event={"ID":"5fc0e383-1643-48e2-ac95-ecb52dd72ac3","Type":"ContainerStarted","Data":"286a1121665c398a1c87fe1bbd545772595dc466d31457f65a8fc0361817f0c2"}
Sep 11 21:32:20.076815 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.076754    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dqb5r" podStartSLOduration=35.482935653 podStartE2EDuration="38.076738105s" totalImagesPullingTime="2.593802452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:31:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:20.075647384 +0000 UTC m=+106.038672035" watchObservedRunningTime="2026-09-11 21:32:20.076738105 +0000 UTC m=+106.039762751"
Sep 11 21:32:20.095848 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.095815    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:20.118851 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.118802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.118977 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.118887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.118977 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.118914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jblc\" (UniqueName: \"kubernetes.io/projected/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-kube-api-access-2jblc\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.118977 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.118963    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.119689 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.119663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.121482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.121459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.121581 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.121458    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.127569 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.127542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jblc\" (UniqueName: \"kubernetes.io/projected/bbda4e59-3f20-4efa-bb14-cccfcae0dda9-kube-api-access-2jblc\") pod \"prometheus-operator-68bcd7f987-q99jd\" (UID: \"bbda4e59-3f20-4efa-bb14-cccfcae0dda9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.233248 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.233219    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"
Sep 11 21:32:20.501392 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:20.501350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-q99jd"]
Sep 11 21:32:20.504366 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:20.504337    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbda4e59_3f20_4efa_bb14_cccfcae0dda9.slice/crio-f61c4355e713e708419bc626f3369793b038d86634ffecbb7724e670ed694766 WatchSource:0}: Error finding container f61c4355e713e708419bc626f3369793b038d86634ffecbb7724e670ed694766: Status 404 returned error can't find the container with id f61c4355e713e708419bc626f3369793b038d86634ffecbb7724e670ed694766
Sep 11 21:32:21.056590 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:21.056552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:21.056819 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:21.056621    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:21.057841 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:21.057812    2582 patch_prober.go:28] interesting pod/console-f8c48665c-x5h7m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body=
Sep 11 21:32:21.058212 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:21.057863    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-f8c48665c-x5h7m" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerName="console" probeResult="failure" output="Get \"https://10.132.0.14:8443/health\": dial tcp 10.132.0.14:8443: connect: connection refused"
Sep 11 21:32:21.062191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:21.062159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd" event={"ID":"bbda4e59-3f20-4efa-bb14-cccfcae0dda9","Type":"ContainerStarted","Data":"f61c4355e713e708419bc626f3369793b038d86634ffecbb7724e670ed694766"}
Sep 11 21:32:22.054195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:22.054103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:22.067477 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:22.067452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd" event={"ID":"bbda4e59-3f20-4efa-bb14-cccfcae0dda9","Type":"ContainerStarted","Data":"9e25283158f058330c1cac167e605dee8f1a83697dd3dc2e3d186edd46cd0302"}
Sep 11 21:32:22.067844 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:22.067509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd" event={"ID":"bbda4e59-3f20-4efa-bb14-cccfcae0dda9","Type":"ContainerStarted","Data":"cd8a34f4ba307566b9d4983e969139f643860adb0541dee33fcbc96a9cfde283"}
Sep 11 21:32:22.086679 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:22.086631    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-q99jd" podStartSLOduration=1.805441487 podStartE2EDuration="3.086616056s" totalImagesPullingTime="1.281174569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:22.083332581 +0000 UTC m=+108.046357230" watchObservedRunningTime="2026-09-11 21:32:22.086616056 +0000 UTC m=+108.049640750"
Sep 11 21:32:23.126119 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:23.126080    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:23.181497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:23.181465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.023965 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.023905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.237742 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.237678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.292482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.292395    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8hl4k"]
Sep 11 21:32:24.294523 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.294506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.296735 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.296688    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:32:24.296735 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.296700    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:32:24.297071 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.297054    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b8zbj\""
Sep 11 21:32:24.297149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.297111    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:32:24.369884 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.369846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.453632 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-textfile\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-sys\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-wtmp\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tqk\" (UniqueName: \"kubernetes.io/projected/aba9dd1c-1f2d-4969-8a68-459379963079-kube-api-access-w2tqk\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-root\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.453971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.453937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-metrics-client-ca\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.504589 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.504546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.555380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-metrics-client-ca\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-textfile\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:32:24.555437    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-sys\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:32:24.555510    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls podName:aba9dd1c-1f2d-4969-8a68-459379963079 nodeName:}" failed. No retries permitted until 2026-09-11 21:32:25.055493177 +0000 UTC m=+111.018517808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls") pod "node-exporter-8hl4k" (UID: "aba9dd1c-1f2d-4969-8a68-459379963079") : secret "node-exporter-tls" not found
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-wtmp\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tqk\" (UniqueName: \"kubernetes.io/projected/aba9dd1c-1f2d-4969-8a68-459379963079-kube-api-access-w2tqk\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-root\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-sys\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-root\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.555953 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.555732    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-wtmp\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.556210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.556076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-textfile\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.556210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.556111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.556210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.556140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aba9dd1c-1f2d-4969-8a68-459379963079-metrics-client-ca\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.558156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.558135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.565589 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.565565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tqk\" (UniqueName: \"kubernetes.io/projected/aba9dd1c-1f2d-4969-8a68-459379963079-kube-api-access-w2tqk\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:24.603218 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.603183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.703670 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.703631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.829119 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.829030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:24.926427 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:24.926384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.059827 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.059785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:25.062353 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.062320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aba9dd1c-1f2d-4969-8a68-459379963079-node-exporter-tls\") pod \"node-exporter-8hl4k\" (UID: \"aba9dd1c-1f2d-4969-8a68-459379963079\") " pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:25.189146 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.189050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.204968 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.204935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hl4k"
Sep 11 21:32:25.215734 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:25.215695    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba9dd1c_1f2d_4969_8a68_459379963079.slice/crio-ae6437e8ae11f52cfae5318502c7ba189b57bb62ea230096ed35ba46515d7b26 WatchSource:0}: Error finding container ae6437e8ae11f52cfae5318502c7ba189b57bb62ea230096ed35ba46515d7b26: Status 404 returned error can't find the container with id ae6437e8ae11f52cfae5318502c7ba189b57bb62ea230096ed35ba46515d7b26
Sep 11 21:32:25.298338 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.298299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.371526 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.371480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.499795 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.499752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.681316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.681280    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:25.738735 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:25.738691    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:26.082634 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.082602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hl4k" event={"ID":"aba9dd1c-1f2d-4969-8a68-459379963079","Type":"ContainerStarted","Data":"ae6437e8ae11f52cfae5318502c7ba189b57bb62ea230096ed35ba46515d7b26"}
Sep 11 21:32:26.296154 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.296127    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65f5c67547-4wq9w"]
Sep 11 21:32:26.299124 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.299108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.301384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 11 21:32:26.301481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301398    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 11 21:32:26.301481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301357    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 11 21:32:26.301481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301379    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tnqph\""
Sep 11 21:32:26.301631 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301398    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 11 21:32:26.301631 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d1adm0d6j16j6\""
Sep 11 21:32:26.301631 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.301424    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 11 21:32:26.310227 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.310199    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65f5c67547-4wq9w"]
Sep 11 21:32:26.471214 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487e1406-1338-4a87-b700-d28debb5ffcc-metrics-client-ca\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471532 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqp8p\" (UniqueName: \"kubernetes.io/projected/487e1406-1338-4a87-b700-d28debb5ffcc-kube-api-access-rqp8p\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471532 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.471532 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.471489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-grpc-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572534 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqp8p\" (UniqueName: \"kubernetes.io/projected/487e1406-1338-4a87-b700-d28debb5ffcc-kube-api-access-rqp8p\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572809    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572850    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-grpc-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572916    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.572954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.572946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487e1406-1338-4a87-b700-d28debb5ffcc-metrics-client-ca\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.573974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.573945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/487e1406-1338-4a87-b700-d28debb5ffcc-metrics-client-ca\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.575564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.575541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.575674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.575566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.575919 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.575897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.575985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.575963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.576067 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.576054    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-grpc-tls\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.576450 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.576427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/487e1406-1338-4a87-b700-d28debb5ffcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.581660 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.581626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqp8p\" (UniqueName: \"kubernetes.io/projected/487e1406-1338-4a87-b700-d28debb5ffcc-kube-api-access-rqp8p\") pod \"thanos-querier-65f5c67547-4wq9w\" (UID: \"487e1406-1338-4a87-b700-d28debb5ffcc\") " pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.615397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.615313    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:26.892516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:26.892438    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65f5c67547-4wq9w"]
Sep 11 21:32:26.895885 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:26.895859    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487e1406_1338_4a87_b700_d28debb5ffcc.slice/crio-d3181e1e11c3969feb3a341bc3b762563ca6eee599dbec75c6d6ea657cf09c57 WatchSource:0}: Error finding container d3181e1e11c3969feb3a341bc3b762563ca6eee599dbec75c6d6ea657cf09c57: Status 404 returned error can't find the container with id d3181e1e11c3969feb3a341bc3b762563ca6eee599dbec75c6d6ea657cf09c57
Sep 11 21:32:27.048136 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.048108    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r82z5"
Sep 11 21:32:27.088156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.088125    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"d3181e1e11c3969feb3a341bc3b762563ca6eee599dbec75c6d6ea657cf09c57"}
Sep 11 21:32:27.089875 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.089845    2582 generic.go:356] "Generic (PLEG): container finished" podID="aba9dd1c-1f2d-4969-8a68-459379963079" containerID="5527224564d52c19d810a8b4a9cf3520a4130177102d14c98ebd1d1cd5786fce" exitCode=0
Sep 11 21:32:27.089993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.089889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hl4k" event={"ID":"aba9dd1c-1f2d-4969-8a68-459379963079","Type":"ContainerDied","Data":"5527224564d52c19d810a8b4a9cf3520a4130177102d14c98ebd1d1cd5786fce"}
Sep 11 21:32:27.220033 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.220006    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:27.318729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.318655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:27.440275 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:27.440185    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:28.096791 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:28.096751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hl4k" event={"ID":"aba9dd1c-1f2d-4969-8a68-459379963079","Type":"ContainerStarted","Data":"b881a5e049c43dcb049ad83a382b1895f85aaf2066af3d7e5c4212faf6556de2"}
Sep 11 21:32:28.096991 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:28.096815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hl4k" event={"ID":"aba9dd1c-1f2d-4969-8a68-459379963079","Type":"ContainerStarted","Data":"4ee211d2400f32293b9b9f3899deab73882bd9f0e51923b0760a35958e0f486b"}
Sep 11 21:32:28.118357 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:28.118298    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8hl4k" podStartSLOduration=2.989113417 podStartE2EDuration="4.118281178s" totalImagesPullingTime="1.057787755s" totalInitContainerRuntime="71.380006ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:28.11650778 +0000 UTC m=+114.079532443" watchObservedRunningTime="2026-09-11 21:32:28.118281178 +0000 UTC m=+114.081305826"
Sep 11 21:32:29.080283 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.080193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:29.103222 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.103195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"2a62d821feb3123fa785d1b6d456ef626cbda4d6f49f73788a91421968392d5a"}
Sep 11 21:32:29.103324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.103277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"491750d525fbc0501c858364d1b9033b2393bcdf926d426578e966a6773459c7"}
Sep 11 21:32:29.146273 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.146247    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:29.152992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.152951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:29.225635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:29.225539    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:30.114118 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.114085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"db1b54e19fc578cb3a5dd9b1c1f79690dfe20959016908619142b31a8fc11a6a"}
Sep 11 21:32:30.114423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.114172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"931a897ecdecfb7ea51ac33ee43e5a5c15523b9dec5018ab73c9fb0239eaeb7e"}
Sep 11 21:32:30.467419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.467328    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:30.472780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.472754    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:32:30.475824 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.475800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.478773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.478748    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 21:32:30.479506 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.479481    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 21:32:30.480488 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480463    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 21:32:30.480599 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480480    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 21:32:30.480599 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480546    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 21:32:30.480952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 21:32:30.480952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1e6lsp2n6b041\""
Sep 11 21:32:30.480952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480814    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 21:32:30.480952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.480861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 21:32:30.481210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.481030    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 21:32:30.481263 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.481121    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 21:32:30.483437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.481876    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 21:32:30.483437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.481930    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 21:32:30.483437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.482085    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4vssb\""
Sep 11 21:32:30.483437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.482295    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 21:32:30.483437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.482521    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 21:32:30.484211 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.484193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 21:32:30.498041 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.498008    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:32:30.610063 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610531 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.610763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.610758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrkn\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-kube-api-access-rhrkn\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711786 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711786 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.711905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.711967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrkn\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-kube-api-access-rhrkn\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712600 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.712600 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.712318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.713704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.713394    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.713845 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.713780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.714504 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.714481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.714946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.714879    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.715233 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.715207    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.715749 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.715705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.717519 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.717459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.717665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.717907    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-web-config\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.718051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.718067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.718289    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5db98f50-d6e5-4b54-9aad-a6f444826bfd-config-out\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.718960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.718329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.719249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.719219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.719389 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.719358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.719787 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.719760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5db98f50-d6e5-4b54-9aad-a6f444826bfd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.719918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.719815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.719918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.719893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5db98f50-d6e5-4b54-9aad-a6f444826bfd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.721573 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.721552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrkn\" (UniqueName: \"kubernetes.io/projected/5db98f50-d6e5-4b54-9aad-a6f444826bfd-kube-api-access-rhrkn\") pod \"prometheus-k8s-0\" (UID: \"5db98f50-d6e5-4b54-9aad-a6f444826bfd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:30.790562 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:30.790520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:31.071136 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.071017    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 21:32:31.073465 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:31.073437    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db98f50_d6e5_4b54_9aad_a6f444826bfd.slice/crio-aca33c674294eecfa77020eb4802c5bfdcc5d137b684445a3a14f35e9702c7a8 WatchSource:0}: Error finding container aca33c674294eecfa77020eb4802c5bfdcc5d137b684445a3a14f35e9702c7a8: Status 404 returned error can't find the container with id aca33c674294eecfa77020eb4802c5bfdcc5d137b684445a3a14f35e9702c7a8
Sep 11 21:32:31.121091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.121050    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"85272614a9bcd7cf25c897b1e8de7d4e648e70827fe8853e8a451a110da5400d"}
Sep 11 21:32:31.121564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.121131    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" event={"ID":"487e1406-1338-4a87-b700-d28debb5ffcc","Type":"ContainerStarted","Data":"c6689941228b50cfcb5509d8437d672ab10b30974c0c18a24ade2550e5b7ea3f"}
Sep 11 21:32:31.121564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.121196    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:31.122278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.122258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"aca33c674294eecfa77020eb4802c5bfdcc5d137b684445a3a14f35e9702c7a8"}
Sep 11 21:32:31.142218 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:31.142162    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w" podStartSLOduration=2.207620214 podStartE2EDuration="5.142144127s" totalImagesPullingTime="2.934523913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:31.141080762 +0000 UTC m=+117.104105411" watchObservedRunningTime="2026-09-11 21:32:31.142144127 +0000 UTC m=+117.105168855"
Sep 11 21:32:32.131049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.131010    2582 generic.go:356] "Generic (PLEG): container finished" podID="5db98f50-d6e5-4b54-9aad-a6f444826bfd" containerID="e6a8371d496f222fb980bb5ac240f0c81179ad435e15cd80b07df10ab5e91251" exitCode=0
Sep 11 21:32:32.131520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.131096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerDied","Data":"e6a8371d496f222fb980bb5ac240f0c81179ad435e15cd80b07df10ab5e91251"}
Sep 11 21:32:32.293596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.293512    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:32:32.298619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.298584    2582 patch_prober.go:28] interesting pod/image-registry-69dc887c94-68psq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 11 21:32:32.298803 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.298647    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69dc887c94-68psq" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:32:32.379333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.379299    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:32.477426 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.477391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:32.551935 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:32.551839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:33.766127 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:33.766091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:33.818091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:33.818032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:34.610731 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:34.610666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:34.674732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:34.674683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:36.150298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150258    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"3bffb2f8b449acc44a4c34337f3143ec32000e8cec9e91f4eacdd5f34ea70393"}
Sep 11 21:32:36.150647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"d10fd9c074af98c52391a92b9f41748b30c5907bdc325375a95160f4fe2b0e20"}
Sep 11 21:32:36.150647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"805041b5be1ffd926864c3b5f8ef0b47ab1e78ec97bc8b892f4b6f438e8d0bbd"}
Sep 11 21:32:36.150647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"f4fa89a8122ce18aedf4dfb2cac4cc7437ca7d9dad3225f1c08367d59b45ec48"}
Sep 11 21:32:36.150647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150456    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"b0d802f6bf3b2b01e77df6959b105fde838ddbe0278d2f75f1644e3471cacb03"}
Sep 11 21:32:36.150647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.150490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5db98f50-d6e5-4b54-9aad-a6f444826bfd","Type":"ContainerStarted","Data":"39dc62a8ec2a09bd7e9144e9ffc7778d6329669d1ccacdb2b7270bee7d309a36"}
Sep 11 21:32:36.181039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.180985    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.993095454 podStartE2EDuration="6.180967818s" totalImagesPullingTime="4.171462056s" totalInitContainerRuntime="16.410308ms" isStatefulPod=false podCreationTimestamp="2026-09-11 21:32:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:36.178833829 +0000 UTC m=+122.141858494" watchObservedRunningTime="2026-09-11 21:32:36.180967818 +0000 UTC m=+122.143992466"
Sep 11 21:32:36.198278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.198236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:36.254255 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:36.254221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:37.138744 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:37.138594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65f5c67547-4wq9w"
Sep 11 21:32:38.537122 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:38.537084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:38.608733 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:38.608689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:40.116088 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.116057    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:40.180564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.180527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:40.685612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.685575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:40.785020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.784987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:40.790807 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.790785    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:32:40.853862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:40.853826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:42.298105 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:42.298077    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:42.785916 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:42.785882    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:42.886291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:42.886260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:44.336754 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:44.336695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:32:44.339072 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:44.339055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd4f841-9875-4dbd-be36-56e9a9bd205d-metrics-certs\") pod \"network-metrics-daemon-x7x2x\" (UID: \"4dd4f841-9875-4dbd-be36-56e9a9bd205d\") " pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:32:44.375514 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:44.375489    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r57v4\""
Sep 11 21:32:44.383267 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:44.383245    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7x2x"
Sep 11 21:32:44.662823 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:44.662742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7x2x"]
Sep 11 21:32:44.666170 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:32:44.666132    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd4f841_9875_4dbd_be36_56e9a9bd205d.slice/crio-39a2010f44fad81675905616f708de3967e031b8e373b97c630259288451a7d2 WatchSource:0}: Error finding container 39a2010f44fad81675905616f708de3967e031b8e373b97c630259288451a7d2: Status 404 returned error can't find the container with id 39a2010f44fad81675905616f708de3967e031b8e373b97c630259288451a7d2
Sep 11 21:32:45.188635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:45.188597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7x2x" event={"ID":"4dd4f841-9875-4dbd-be36-56e9a9bd205d","Type":"ContainerStarted","Data":"39a2010f44fad81675905616f708de3967e031b8e373b97c630259288451a7d2"}
Sep 11 21:32:47.199484 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.199446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7x2x" event={"ID":"4dd4f841-9875-4dbd-be36-56e9a9bd205d","Type":"ContainerStarted","Data":"808fa54ba3faa913fac706afe0ecb1e9c71ab26e32f1ace752fa51ee02d4d1d6"}
Sep 11 21:32:47.199898 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.199500    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7x2x" event={"ID":"4dd4f841-9875-4dbd-be36-56e9a9bd205d","Type":"ContainerStarted","Data":"07f2d9ff41110e73959750aac787bb2b0cd490dcf01ad59554beb43fc3c9add5"}
Sep 11 21:32:47.218193 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.218135    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x7x2x" podStartSLOduration=130.554720922 podStartE2EDuration="2m12.218119775s" totalImagesPullingTime="1.663398853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:32:47.217055339 +0000 UTC m=+133.180080010" watchObservedRunningTime="2026-09-11 21:32:47.218119775 +0000 UTC m=+133.181144426"
Sep 11 21:32:47.379726 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.379685    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:47.480954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.480920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:47.546257 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.546214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:47.640700 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.640660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:47.744775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:47.744669    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:50.287815 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:50.287780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:50.353869 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:50.353833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:54.166419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.166377    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f8c48665c-x5h7m" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerName="console" containerID="cri-o://29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db" gracePeriod=15
Sep 11 21:32:54.214372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.214285    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:54.416680 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.416625    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:54.530409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmpjd\" (UniqueName: \"kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530838 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert\") pod \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\" (UID: \"f57ac5dd-28ab-4e7a-9997-3d9c03686932\") "
Sep 11 21:32:54.530971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530944    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:54.530971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530950    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config" (OuterVolumeSpecName: "console-config") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:54.531071 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.530968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca" (OuterVolumeSpecName: "service-ca") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:54.533101 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.533067    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:32:54.533203 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.533097    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:32:54.533203 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.533126    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd" (OuterVolumeSpecName: "kube-api-access-wmpjd") pod "f57ac5dd-28ab-4e7a-9997-3d9c03686932" (UID: "f57ac5dd-28ab-4e7a-9997-3d9c03686932"). InnerVolumeSpecName "kube-api-access-wmpjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:54.631710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631668    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-serving-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:54.631710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631699    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-oauth-config\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:54.631710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631710    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpjd\" (UniqueName: \"kubernetes.io/projected/f57ac5dd-28ab-4e7a-9997-3d9c03686932-kube-api-access-wmpjd\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:54.631969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631754    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-console-config\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:54.631969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631767    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-oauth-serving-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:54.631969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:54.631779    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f57ac5dd-28ab-4e7a-9997-3d9c03686932-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:55.044960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.044920    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:55.145801 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.145760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:55.232168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.232137    2582 generic.go:356] "Generic (PLEG): container finished" podID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerID="29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db" exitCode=0
Sep 11 21:32:55.232540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.232179    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8c48665c-x5h7m"
Sep 11 21:32:55.232540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.232226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c48665c-x5h7m" event={"ID":"f57ac5dd-28ab-4e7a-9997-3d9c03686932","Type":"ContainerDied","Data":"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"}
Sep 11 21:32:55.232540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.232277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8c48665c-x5h7m" event={"ID":"f57ac5dd-28ab-4e7a-9997-3d9c03686932","Type":"ContainerDied","Data":"21cf6a979232eade610ea5ff76e6d13ad720e2d775c428db7797f20e313a6e9e"}
Sep 11 21:32:55.232540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.232315    2582 scope.go:117] "RemoveContainer" containerID="29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"
Sep 11 21:32:55.241311 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.241291    2582 scope.go:117] "RemoveContainer" containerID="29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"
Sep 11 21:32:55.241594 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:32:55.241574    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db\": container with ID starting with 29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db not found: ID does not exist" containerID="29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"
Sep 11 21:32:55.241652 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.241603    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db"} err="failed to get container status \"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db\": rpc error: code = NotFound desc = could not find container \"29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db\": container with ID starting with 29399ef38a7991ca15bc0bed68ce3a26ac8053a483356f0f840e2727978204db not found: ID does not exist"
Sep 11 21:32:55.252506 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.252478    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:55.254873 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:55.254852    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8c48665c-x5h7m"]
Sep 11 21:32:56.453654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:56.453620    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" path="/var/lib/kubelet/pods/f57ac5dd-28ab-4e7a-9997-3d9c03686932/volumes"
Sep 11 21:32:57.316412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.316366    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69dc887c94-68psq" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerName="registry" containerID="cri-o://826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1" gracePeriod=30
Sep 11 21:32:57.555972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.555950    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:57.655100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655059    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655366 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655123    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655366 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655366 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655217    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655366 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655350    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmtm\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655558 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates\") pod \"03dd5e85-1079-460e-acf5-d5df9bac46bc\" (UID: \"03dd5e85-1079-460e-acf5-d5df9bac46bc\") "
Sep 11 21:32:57.655558 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:57.656079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.655882    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-trusted-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.656331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.656305    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:32:57.657872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.657823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:32:57.657989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.657931    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:32:57.657989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.657949    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm" (OuterVolumeSpecName: "kube-api-access-jgmtm") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "kube-api-access-jgmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:57.658079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.658061    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:57.658398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.658379    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:32:57.664458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.664434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "03dd5e85-1079-460e-acf5-d5df9bac46bc" (UID: "03dd5e85-1079-460e-acf5-d5df9bac46bc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:32:57.704832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.704788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:57.756390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756362    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03dd5e85-1079-460e-acf5-d5df9bac46bc-ca-trust-extracted\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756386    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756397    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmtm\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-kube-api-access-jgmtm\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756406    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03dd5e85-1079-460e-acf5-d5df9bac46bc-registry-certificates\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756414    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-installation-pull-secrets\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756423    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03dd5e85-1079-460e-acf5-d5df9bac46bc-bound-sa-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.756567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.756432    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03dd5e85-1079-460e-acf5-d5df9bac46bc-image-registry-private-configuration\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:32:57.804046 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:57.804017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:32:58.246766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.246705    2582 generic.go:356] "Generic (PLEG): container finished" podID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerID="826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1" exitCode=0
Sep 11 21:32:58.246766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.246753    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dc887c94-68psq" event={"ID":"03dd5e85-1079-460e-acf5-d5df9bac46bc","Type":"ContainerDied","Data":"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"}
Sep 11 21:32:58.246992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.246807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69dc887c94-68psq" event={"ID":"03dd5e85-1079-460e-acf5-d5df9bac46bc","Type":"ContainerDied","Data":"6bebfce850b58b7718ffc263d73626a7a975393e8669a4551b861549751413f5"}
Sep 11 21:32:58.246992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.246827    2582 scope.go:117] "RemoveContainer" containerID="826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"
Sep 11 21:32:58.246992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.246765    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69dc887c94-68psq"
Sep 11 21:32:58.255099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.255082    2582 scope.go:117] "RemoveContainer" containerID="826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"
Sep 11 21:32:58.255354 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:32:58.255339    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1\": container with ID starting with 826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1 not found: ID does not exist" containerID="826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"
Sep 11 21:32:58.255397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.255363    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1"} err="failed to get container status \"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1\": rpc error: code = NotFound desc = could not find container \"826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1\": container with ID starting with 826c47038d9f9bc1649fb579242d8752c4355031ea0a2f0cff507c23ef1587c1 not found: ID does not exist"
Sep 11 21:32:58.267166 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.267140    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:32:58.269300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.269279    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69dc887c94-68psq"]
Sep 11 21:32:58.453766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:32:58.453733    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" path="/var/lib/kubelet/pods/03dd5e85-1079-460e-acf5-d5df9bac46bc/volumes"
Sep 11 21:33:02.264022 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:02.263986    2582 generic.go:356] "Generic (PLEG): container finished" podID="24d43615-b6a3-40ff-a376-4c095814d988" containerID="585110c616c4ab9f355e899c39c35e05fb45fe2b43ea3038c7bbce12308c8c35" exitCode=0
Sep 11 21:33:02.264452 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:02.264062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wv46w" event={"ID":"24d43615-b6a3-40ff-a376-4c095814d988","Type":"ContainerDied","Data":"585110c616c4ab9f355e899c39c35e05fb45fe2b43ea3038c7bbce12308c8c35"}
Sep 11 21:33:02.264452 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:02.264429    2582 scope.go:117] "RemoveContainer" containerID="585110c616c4ab9f355e899c39c35e05fb45fe2b43ea3038c7bbce12308c8c35"
Sep 11 21:33:03.269768 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:03.269710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-wv46w" event={"ID":"24d43615-b6a3-40ff-a376-4c095814d988","Type":"ContainerStarted","Data":"56af9ff42ffd7c421834ff7a877c5399dec7ded994e0247c38399cebaa44d158"}
Sep 11 21:33:07.287451 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:07.287415    2582 generic.go:356] "Generic (PLEG): container finished" podID="86072044-4771-4739-9eb1-27a0963b134e" containerID="1e7f96600c095934ac3ea1cfd7794e3932ab345d62c3112b00ddda02e73717d5" exitCode=0
Sep 11 21:33:07.287848 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:07.287494    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj" event={"ID":"86072044-4771-4739-9eb1-27a0963b134e","Type":"ContainerDied","Data":"1e7f96600c095934ac3ea1cfd7794e3932ab345d62c3112b00ddda02e73717d5"}
Sep 11 21:33:07.287896 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:07.287854    2582 scope.go:117] "RemoveContainer" containerID="1e7f96600c095934ac3ea1cfd7794e3932ab345d62c3112b00ddda02e73717d5"
Sep 11 21:33:08.293448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:08.293417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xnhpj" event={"ID":"86072044-4771-4739-9eb1-27a0963b134e","Type":"ContainerStarted","Data":"cf32d85c13e122c2f9a61b6fcb1362a3c1426f971ce30cab043e609fcf5b0644"}
Sep 11 21:33:10.052475 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:10.052440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:10.111555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:10.111520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:30.790703 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:30.790619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:33:30.810905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:30.810877    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:33:31.401964 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:31.401938    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 21:33:43.490463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:43.490408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:43.885989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:43.885881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:51.977891 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:51.977854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:58.741620 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:58.741546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:33:58.802150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:33:58.802113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:10.588602 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:10.588538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:10.660401 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:10.660369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:14.069519 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:14.069479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:14.131540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:14.131504    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:14.192545 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:14.192502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:18.811112 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:18.811081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:18.912939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:18.912907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:33.059729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:33.059678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:33.161447 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:33.161416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:39.133540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:39.133488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:52.340587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:52.340531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:58.631084 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:58.630986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:34:58.728904 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:34:58.728861    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:20.393382 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:20.393343    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:21.179871 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:21.179832    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:23.994780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:23.994744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:24.095372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:24.095332    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:25.465580 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:25.465548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:25.589898 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:25.589867    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:25.716226 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:25.716136    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:27.899836 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:27.899809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:28.116179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:28.116134    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:30.989974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:30.989933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:31.091246 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:31.091214    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:34.396599 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:34.396574    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:35:35.712783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:35.712747    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:36.448187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448155    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zk9jh"]
Sep 11 21:35:36.448456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448444    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerName="registry"
Sep 11 21:35:36.448517 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448457    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerName="registry"
Sep 11 21:35:36.448517 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448481    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerName="console"
Sep 11 21:35:36.448517 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448486    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerName="console"
Sep 11 21:35:36.448629 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448538    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03dd5e85-1079-460e-acf5-d5df9bac46bc" containerName="registry"
Sep 11 21:35:36.448629 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.448547    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57ac5dd-28ab-4e7a-9997-3d9c03686932" containerName="console"
Sep 11 21:35:36.451052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.451025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.453195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.453174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 21:35:36.453297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.453174    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 21:35:36.453825 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.453804    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tt6ns\""
Sep 11 21:35:36.458790 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.458773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zk9jh"]
Sep 11 21:35:36.521690 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.521657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-bound-sa-token\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.521881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.521735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdmj\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-kube-api-access-mjdmj\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.622825 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.622791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-bound-sa-token\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.623006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.622871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdmj\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-kube-api-access-mjdmj\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.630398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.630366    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-bound-sa-token\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.630655 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.630634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdmj\" (UniqueName: \"kubernetes.io/projected/7adbab9e-4104-41de-b276-40e574e68bf2-kube-api-access-mjdmj\") pod \"cert-manager-55b4d954d5-zk9jh\" (UID: \"7adbab9e-4104-41de-b276-40e574e68bf2\") " pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:36.761826 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:36.761796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zk9jh"
Sep 11 21:35:37.030481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:37.030402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zk9jh"]
Sep 11 21:35:37.033193 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:35:37.033166    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7adbab9e_4104_41de_b276_40e574e68bf2.slice/crio-73e9f01f7c9501d398bfca0c1a94f51d5832d24c4679fa2c88a3cf166cbd2799 WatchSource:0}: Error finding container 73e9f01f7c9501d398bfca0c1a94f51d5832d24c4679fa2c88a3cf166cbd2799: Status 404 returned error can't find the container with id 73e9f01f7c9501d398bfca0c1a94f51d5832d24c4679fa2c88a3cf166cbd2799
Sep 11 21:35:37.035078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:37.035062    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:35:37.854281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:37.854234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zk9jh" event={"ID":"7adbab9e-4104-41de-b276-40e574e68bf2","Type":"ContainerStarted","Data":"73e9f01f7c9501d398bfca0c1a94f51d5832d24c4679fa2c88a3cf166cbd2799"}
Sep 11 21:35:38.475302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:38.475243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:38.576516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:38.576369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:40.166577 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.166542    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"]
Sep 11 21:35:40.170890 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.170857    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.173326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.173304    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 21:35:40.173469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.173328    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dgs24\""
Sep 11 21:35:40.173880 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.173861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:35:40.179437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.179406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"]
Sep 11 21:35:40.254548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.254517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887kr\" (UniqueName: \"kubernetes.io/projected/24c59373-aeb4-49d2-aade-396c566b7143-kube-api-access-887kr\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.254548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.254561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c59373-aeb4-49d2-aade-396c566b7143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.355553 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.355515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887kr\" (UniqueName: \"kubernetes.io/projected/24c59373-aeb4-49d2-aade-396c566b7143-kube-api-access-887kr\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.355680 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.355564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c59373-aeb4-49d2-aade-396c566b7143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.356003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.355985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c59373-aeb4-49d2-aade-396c566b7143-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.364579 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.364555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887kr\" (UniqueName: \"kubernetes.io/projected/24c59373-aeb4-49d2-aade-396c566b7143-kube-api-access-887kr\") pod \"openshift-lws-operator-fd8ccff4c-mnnfk\" (UID: \"24c59373-aeb4-49d2-aade-396c566b7143\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.480654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.480614    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"
Sep 11 21:35:40.752482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.752456    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk"]
Sep 11 21:35:40.755017 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:35:40.754993    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24c59373_aeb4_49d2_aade_396c566b7143.slice/crio-a8702a789dba53fd625edfc7bd61b149fa443777fa84f9d2021951d9fa1cd4f8 WatchSource:0}: Error finding container a8702a789dba53fd625edfc7bd61b149fa443777fa84f9d2021951d9fa1cd4f8: Status 404 returned error can't find the container with id a8702a789dba53fd625edfc7bd61b149fa443777fa84f9d2021951d9fa1cd4f8
Sep 11 21:35:40.867348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.867306    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk" event={"ID":"24c59373-aeb4-49d2-aade-396c566b7143","Type":"ContainerStarted","Data":"a8702a789dba53fd625edfc7bd61b149fa443777fa84f9d2021951d9fa1cd4f8"}
Sep 11 21:35:40.868806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.868777    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zk9jh" event={"ID":"7adbab9e-4104-41de-b276-40e574e68bf2","Type":"ContainerStarted","Data":"2a7ce2fef364af5accf8fdd513821185fe3dd5fe5dd17a5df66bd114eba573a6"}
Sep 11 21:35:40.885121 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:40.885076    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zk9jh" podStartSLOduration=1.9913845540000001 podStartE2EDuration="4.885063461s" totalImagesPullingTime="2.893678907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:35:40.884066985 +0000 UTC m=+306.847091842" watchObservedRunningTime="2026-09-11 21:35:40.885063461 +0000 UTC m=+306.848088109"
Sep 11 21:35:43.883882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:43.883850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk" event={"ID":"24c59373-aeb4-49d2-aade-396c566b7143","Type":"ContainerStarted","Data":"969235f24a563d2d2ef25cc8ad9c088267963622de5e17ca342ad3bf95622a79"}
Sep 11 21:35:43.899920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:43.899868    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mnnfk" podStartSLOduration=1.276431472 podStartE2EDuration="3.899851655s" totalImagesPullingTime="2.623420183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:35:43.898636768 +0000 UTC m=+309.861661421" watchObservedRunningTime="2026-09-11 21:35:43.899851655 +0000 UTC m=+309.862876314"
Sep 11 21:35:44.585671 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:44.585640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:44.686842 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:44.686809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:46.931074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:46.931028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:35:59.115306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:35:59.115273    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:00.346306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:00.346268    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:02.230327 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:02.230284    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:02.345522 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:02.345488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:02.658436 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:02.658360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:02.940987 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:02.940898    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.042512 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.042473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.128685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.128648    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"]
Sep 11 21:36:03.182649 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.182617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"]
Sep 11 21:36:03.182857 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.182779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.185869 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.185839    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:36:03.186128 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.186109    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:36:03.186196 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.186133    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:36:03.186382 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.186359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vmt8t\""
Sep 11 21:36:03.186584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.186566    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:36:03.242262 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.242230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.242654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.242294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.242654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.242343    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcrp\" (UniqueName: \"kubernetes.io/projected/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-kube-api-access-htcrp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.242654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.242406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.242654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.242439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.310434 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.310390    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.343237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.343237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.343439 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.343439 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.343525 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcrp\" (UniqueName: \"kubernetes.io/projected/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-kube-api-access-htcrp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.343821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.343800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-manifests\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.345654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.345636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-tmp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.346066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.346047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.346123 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.346047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-webhook-cert\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.351304 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.351283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcrp\" (UniqueName: \"kubernetes.io/projected/4073f1b0-19ff-4a45-bfa5-e817f0a14ebe-kube-api-access-htcrp\") pod \"opendatahub-operator-controller-manager-595996979b-rl657\" (UID: \"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe\") " pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.411278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.411221    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.493410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.493324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:03.511592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.511564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.611220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.611186    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.712828 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.712795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:03.806611 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.806564    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"]
Sep 11 21:36:03.809022 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:36:03.808998    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4073f1b0_19ff_4a45_bfa5_e817f0a14ebe.slice/crio-1af1c09f5e6c13bdbf3d76af64566d3faa9e5b61711117652c74883ef7ce24af WatchSource:0}: Error finding container 1af1c09f5e6c13bdbf3d76af64566d3faa9e5b61711117652c74883ef7ce24af: Status 404 returned error can't find the container with id 1af1c09f5e6c13bdbf3d76af64566d3faa9e5b61711117652c74883ef7ce24af
Sep 11 21:36:03.960407 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:03.960369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657" event={"ID":"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe","Type":"ContainerStarted","Data":"1af1c09f5e6c13bdbf3d76af64566d3faa9e5b61711117652c74883ef7ce24af"}
Sep 11 21:36:05.263114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:05.263082    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:05.357473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:05.357334    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:06.977281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:06.977236    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657" event={"ID":"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe","Type":"ContainerStarted","Data":"de46a660b4b88dca6e1779bd7f027b35e5811e47eccb4e71f564eb66148ef40d"}
Sep 11 21:36:08.992929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:08.992894    2582 generic.go:356] "Generic (PLEG): container finished" podID="4073f1b0-19ff-4a45-bfa5-e817f0a14ebe" containerID="de46a660b4b88dca6e1779bd7f027b35e5811e47eccb4e71f564eb66148ef40d" exitCode=0
Sep 11 21:36:08.992929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:08.992933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657" event={"ID":"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe","Type":"ContainerDied","Data":"de46a660b4b88dca6e1779bd7f027b35e5811e47eccb4e71f564eb66148ef40d"}
Sep 11 21:36:09.998669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:09.998635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657" event={"ID":"4073f1b0-19ff-4a45-bfa5-e817f0a14ebe","Type":"ContainerStarted","Data":"87f3d13924953993aff1a1d5d33bf478ab35e2b21f0f74f518ba81bfeadf2925"}
Sep 11 21:36:09.999088 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:09.998885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:10.026252 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:10.026139    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657" podStartSLOduration=2.630186218 podStartE2EDuration="7.026124384s" totalImagesPullingTime="3.314337877s" totalInitContainerRuntime="1.081600289s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:10.025930237 +0000 UTC m=+335.988954888" watchObservedRunningTime="2026-09-11 21:36:10.026124384 +0000 UTC m=+335.989149032"
Sep 11 21:36:21.005253 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:21.005221    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-595996979b-rl657"
Sep 11 21:36:21.157132 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:21.157096    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:21.218914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:21.218880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:22.562302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:22.562264    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:22.665190 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:22.665150    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:22.935832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:22.935743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:23.144911 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:23.144872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:24.938100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:24.938056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:25.056984 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:25.056949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:26.387076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:26.386995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:29.936390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:29.936358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:30.037358 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:30.037317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:30.208976 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:30.208895    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:30.353472 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:30.353434    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:30.579266 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:30.579227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.098257 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.098216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.197479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.197452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.554858 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.554822    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pb4dd"]
Sep 11 21:36:33.556929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.556907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.559094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.559064    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 21:36:33.559187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.559169    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z7z5z\""
Sep 11 21:36:33.571469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.571443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pb4dd"]
Sep 11 21:36:33.596412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.596374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.608616 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.608582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-cert\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.608785 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.608644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzt9\" (UniqueName: \"kubernetes.io/projected/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-kube-api-access-2lzt9\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.686767 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.686730    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.709889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.709858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzt9\" (UniqueName: \"kubernetes.io/projected/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-kube-api-access-2lzt9\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.710039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.709930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-cert\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.712406 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.712383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-cert\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.720064 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.720045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzt9\" (UniqueName: \"kubernetes.io/projected/1ba6e2e3-6730-4cba-a697-e11f9ca8638b-kube-api-access-2lzt9\") pod \"odh-model-controller-7998c67dc4-pb4dd\" (UID: \"1ba6e2e3-6730-4cba-a697-e11f9ca8638b\") " pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.787857 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.787819    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:33.868085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.868004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:33.971653 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.971625    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dm8lc"]
Sep 11 21:36:33.974245 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.974229    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:33.979328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.979300    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 21:36:33.986912 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.986884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dm8lc"]
Sep 11 21:36:33.999895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:33.999868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:34.013325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.013285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69xb\" (UniqueName: \"kubernetes.io/projected/f082e519-3d93-4903-9007-de5ae68d2066-kube-api-access-b69xb\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.013441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.013333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.114249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.114215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69xb\" (UniqueName: \"kubernetes.io/projected/f082e519-3d93-4903-9007-de5ae68d2066-kube-api-access-b69xb\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.114710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.114256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.114710 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:36:34.114370    2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 11 21:36:34.114710 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:36:34.114422    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs podName:f082e519-3d93-4903-9007-de5ae68d2066 nodeName:}" failed. No retries permitted until 2026-09-11 21:36:34.614407787 +0000 UTC m=+360.577432415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs") pod "model-serving-api-54d89bd5fc-dm8lc" (UID: "f082e519-3d93-4903-9007-de5ae68d2066") : secret "model-serving-api-tls" not found
Sep 11 21:36:34.123630 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.123570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69xb\" (UniqueName: \"kubernetes.io/projected/f082e519-3d93-4903-9007-de5ae68d2066-kube-api-access-b69xb\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.138732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.138693    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pb4dd"]
Sep 11 21:36:34.141141 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:36:34.141117    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba6e2e3_6730_4cba_a697_e11f9ca8638b.slice/crio-3a0ea96b7f2858520c8e888ff14f23fc13a0729efeaacf99a10e8a462202af8f WatchSource:0}: Error finding container 3a0ea96b7f2858520c8e888ff14f23fc13a0729efeaacf99a10e8a462202af8f: Status 404 returned error can't find the container with id 3a0ea96b7f2858520c8e888ff14f23fc13a0729efeaacf99a10e8a462202af8f
Sep 11 21:36:34.277082 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.277051    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:34.618285 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.618248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.620702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.620682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f082e519-3d93-4903-9007-de5ae68d2066-tls-certs\") pod \"model-serving-api-54d89bd5fc-dm8lc\" (UID: \"f082e519-3d93-4903-9007-de5ae68d2066\") " pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:34.884420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:34.884334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:35.113724 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:35.113684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd" event={"ID":"1ba6e2e3-6730-4cba-a697-e11f9ca8638b","Type":"ContainerStarted","Data":"3a0ea96b7f2858520c8e888ff14f23fc13a0729efeaacf99a10e8a462202af8f"}
Sep 11 21:36:35.165791 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:35.165670    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dm8lc"]
Sep 11 21:36:35.176975 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:36:35.176443    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf082e519_3d93_4903_9007_de5ae68d2066.slice/crio-b1ea17a05b819a5c2d842ffacf03fca45879aea657dd87736c8940f4d8342274 WatchSource:0}: Error finding container b1ea17a05b819a5c2d842ffacf03fca45879aea657dd87736c8940f4d8342274: Status 404 returned error can't find the container with id b1ea17a05b819a5c2d842ffacf03fca45879aea657dd87736c8940f4d8342274
Sep 11 21:36:36.120889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:36.120850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc" event={"ID":"f082e519-3d93-4903-9007-de5ae68d2066","Type":"ContainerStarted","Data":"b1ea17a05b819a5c2d842ffacf03fca45879aea657dd87736c8940f4d8342274"}
Sep 11 21:36:38.132031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:38.131993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd" event={"ID":"1ba6e2e3-6730-4cba-a697-e11f9ca8638b","Type":"ContainerStarted","Data":"0d01a3ac2e6d1a75e3bb64144ef0e3dcf97478d43947c57e0a2e030f9b8b275c"}
Sep 11 21:36:38.132482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:38.132095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:38.152944 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:38.152893    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd" podStartSLOduration=2.180707733 podStartE2EDuration="5.152875184s" totalImagesPullingTime="2.972167451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:38.15117297 +0000 UTC m=+364.114197611" watchObservedRunningTime="2026-09-11 21:36:38.152875184 +0000 UTC m=+364.115899832"
Sep 11 21:36:39.138478 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:39.138441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc" event={"ID":"f082e519-3d93-4903-9007-de5ae68d2066","Type":"ContainerStarted","Data":"36a94b2986c6e693408a1888eb886bf362dc2e70616a4d4705ecbc45e85eb1b8"}
Sep 11 21:36:39.155658 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:39.155599    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc" podStartSLOduration=2.330173094 podStartE2EDuration="6.155584986s" totalImagesPullingTime="3.825411892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:39.153886846 +0000 UTC m=+365.116911495" watchObservedRunningTime="2026-09-11 21:36:39.155584986 +0000 UTC m=+365.118609634"
Sep 11 21:36:40.143624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.143584    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:40.575078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.575046    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"]
Sep 11 21:36:40.578326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.578309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.582034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.582009    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 21:36:40.582155 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.582045    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 21:36:40.582155 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.582019    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 21:36:40.582155 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.582136    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4mq2d\""
Sep 11 21:36:40.593241 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.593212    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"]
Sep 11 21:36:40.673078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.673041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.673240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.673097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.673240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.673121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmv2\" (UniqueName: \"kubernetes.io/projected/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-kube-api-access-trmv2\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.673240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.673171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.761213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.758771    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"]
Sep 11 21:36:40.763452 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.763422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.766659 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.766634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 21:36:40.766819 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.766746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k9928\""
Sep 11 21:36:40.766966 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.766951    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:36:40.767174 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.767150    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 21:36:40.767174 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.767172    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:36:40.767378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.767201    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 21:36:40.767378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.767215    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:36:40.774622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.774599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.774733 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.774638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.774733 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.774659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmv2\" (UniqueName: \"kubernetes.io/projected/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-kube-api-access-trmv2\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.774733 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.774702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.775283 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.775264    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-manager-config\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.777495 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.777470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-metrics-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.777495 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.777491    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-cert\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.782007 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.781979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"]
Sep 11 21:36:40.785639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.785616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmv2\" (UniqueName: \"kubernetes.io/projected/a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1-kube-api-access-trmv2\") pod \"lws-controller-manager-74f4c6fbc8-twn9t\" (UID: \"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1\") " pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.875799 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzs2\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-kube-api-access-plzs2\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.875946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.875946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.875946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ff451c-9041-4516-a22c-e358c216d096-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.875946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.876078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.876078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.875991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.887676 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.887652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:40.976552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ff451c-9041-4516-a22c-e358c216d096-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzs2\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-kube-api-access-plzs2\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.976973 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.976839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.977579 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.977534    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.979591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.979563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ff451c-9041-4516-a22c-e358c216d096-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.979737 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.979650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.979737 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.979691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.979870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.979854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ff451c-9041-4516-a22c-e358c216d096-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.990188 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.990162    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:40.994179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:40.994158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzs2\" (UniqueName: \"kubernetes.io/projected/a1ff451c-9041-4516-a22c-e358c216d096-kube-api-access-plzs2\") pod \"istiod-openshift-gateway-5677ccb49b-nbn5h\" (UID: \"a1ff451c-9041-4516-a22c-e358c216d096\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:41.072901 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:41.072868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:41.161801 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:41.161778    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"]
Sep 11 21:36:41.163847 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:36:41.163825    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c3e4ec_a12b_4c6b_b4d2_f67f740ba2a1.slice/crio-259b34b4eaa6723761e506cad431aa6d0f8c541de542bc9f7be76d0ca7b53c63 WatchSource:0}: Error finding container 259b34b4eaa6723761e506cad431aa6d0f8c541de542bc9f7be76d0ca7b53c63: Status 404 returned error can't find the container with id 259b34b4eaa6723761e506cad431aa6d0f8c541de542bc9f7be76d0ca7b53c63
Sep 11 21:36:41.394227 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:41.394185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"]
Sep 11 21:36:41.394840 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:36:41.394815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ff451c_9041_4516_a22c_e358c216d096.slice/crio-db0d8073a116b53fbbd593fd92580445fc5da41aa745fed5997c102025decb9b WatchSource:0}: Error finding container db0d8073a116b53fbbd593fd92580445fc5da41aa745fed5997c102025decb9b: Status 404 returned error can't find the container with id db0d8073a116b53fbbd593fd92580445fc5da41aa745fed5997c102025decb9b
Sep 11 21:36:42.156125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:42.156085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h" event={"ID":"a1ff451c-9041-4516-a22c-e358c216d096","Type":"ContainerStarted","Data":"db0d8073a116b53fbbd593fd92580445fc5da41aa745fed5997c102025decb9b"}
Sep 11 21:36:42.159473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:42.159430    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t" event={"ID":"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1","Type":"ContainerStarted","Data":"259b34b4eaa6723761e506cad431aa6d0f8c541de542bc9f7be76d0ca7b53c63"}
Sep 11 21:36:43.165913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.165829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t" event={"ID":"a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1","Type":"ContainerStarted","Data":"7399559c9c24d5af6e7326e0a0365e984a439afa9836aa520e5fedf345129392"}
Sep 11 21:36:43.166281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.165937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:36:43.189763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.189671    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t" podStartSLOduration=1.57014663 podStartE2EDuration="3.189650071s" totalImagesPullingTime="1.619503441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:43.18921229 +0000 UTC m=+369.152236939" watchObservedRunningTime="2026-09-11 21:36:43.189650071 +0000 UTC m=+369.152674721"
Sep 11 21:36:43.767229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.767188    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:43.874906 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.874854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:43.937248 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:43.937209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:44.016485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:44.016446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:44.131249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:44.131128    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:44.299120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:44.299084    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:36:49.141689 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:49.141651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pb4dd"
Sep 11 21:36:49.434724 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:49.434674    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:36:49.434797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:49.434766    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:36:50.201080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:50.201042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h" event={"ID":"a1ff451c-9041-4516-a22c-e358c216d096","Type":"ContainerStarted","Data":"ff2187105535af77301fdcd56f67e84f0ce370bbbca59748f9db448f14fe41ce"}
Sep 11 21:36:50.201518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:50.201210    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:50.202864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:50.202841    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h"
Sep 11 21:36:50.223237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:50.223186    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbn5h" podStartSLOduration=2.185817288 podStartE2EDuration="10.223169422s" totalImagesPullingTime="8.037352134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:36:50.221094325 +0000 UTC m=+376.184118974" watchObservedRunningTime="2026-09-11 21:36:50.223169422 +0000 UTC m=+376.186194071"
Sep 11 21:36:51.154152 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:51.154123    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dm8lc"
Sep 11 21:36:54.172466 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:36:54.172434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74f4c6fbc8-twn9t"
Sep 11 21:37:12.648972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:12.648922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:37:12.750931 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:12.750891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:37:24.112101 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:24.112038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:37:24.217085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:24.217019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:37:34.823829 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:34.823784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:37:34.915727 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:37:34.915694    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:07.332354 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:07.332315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:08.136021 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:08.135984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:11.461507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:11.461465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:11.620837 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:11.620801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:12.992966 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:12.992938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:13.740328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:13.740291    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:14.364338 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:14.364302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:15.730820 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:15.730787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:16.152033 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:16.151941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:16.544870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:16.544835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:17.065415 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:17.065381    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:17.177201 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:17.177166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:21.367046 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:21.367014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:21.463858 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:21.463821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:21.740918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:21.740876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:21.805657 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:21.805618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:24.144123 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:24.144079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:24.243574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:24.243531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:26.984272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:26.984241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:27.363445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:27.363358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:28.478037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:28.478004    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:28.596352 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:28.596320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:29.440260 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:29.440222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:33.197880 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:33.197844    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:33.295520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:33.295483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:33.381245 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:33.381201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:41.243617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:41.243583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:41.342598 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:41.342563    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:41.870851 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:41.870809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:41.986265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:41.986226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:45.621500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:45.621460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:45.737085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:45.737048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:49.425987 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:49.425953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:49.514031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:49.513986    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:53.386734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:53.386680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:53.487081 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:53.487043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:38:53.556734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:38:53.556689    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:03.224442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:03.224402    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:05.924514 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:05.924470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:06.325153 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:06.325119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:17.366237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.366195    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:17.462484 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.462448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:17.569820 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.569780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:17.814062 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.814023    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:39:17.817012 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.816988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:17.819026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.819009    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:39:17.819151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.819126    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 21:39:17.819810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.819784    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:39:17.819895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.819829    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g6m4z\""
Sep 11 21:39:17.825608 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.825584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:39:17.857902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.857872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:39:17.926617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.926583    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94fd6c68-511d-43f8-a0ff-91531fbfc58a-config-file\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:17.926825 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:17.926624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9798\" (UniqueName: \"kubernetes.io/projected/94fd6c68-511d-43f8-a0ff-91531fbfc58a-kube-api-access-p9798\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.027772 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.027730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94fd6c68-511d-43f8-a0ff-91531fbfc58a-config-file\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.027944 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.027787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9798\" (UniqueName: \"kubernetes.io/projected/94fd6c68-511d-43f8-a0ff-91531fbfc58a-kube-api-access-p9798\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.028354 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.028335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94fd6c68-511d-43f8-a0ff-91531fbfc58a-config-file\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.036178 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.036153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9798\" (UniqueName: \"kubernetes.io/projected/94fd6c68-511d-43f8-a0ff-91531fbfc58a-kube-api-access-p9798\") pod \"limitador-limitador-84994f7ffc-bplxp\" (UID: \"94fd6c68-511d-43f8-a0ff-91531fbfc58a\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.127989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.127901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:18.175331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.175293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:18.276509 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.276472    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:18.402950 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.402923    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:39:18.405493 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:39:18.405467    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fd6c68_511d_43f8_a0ff_91531fbfc58a.slice/crio-d90bfb4aafa1bedf9e4b41bf66c588dcbd87e5d79f735884816740b98c1e7c5b WatchSource:0}: Error finding container d90bfb4aafa1bedf9e4b41bf66c588dcbd87e5d79f735884816740b98c1e7c5b: Status 404 returned error can't find the container with id d90bfb4aafa1bedf9e4b41bf66c588dcbd87e5d79f735884816740b98c1e7c5b
Sep 11 21:39:18.875688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:18.875654    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp" event={"ID":"94fd6c68-511d-43f8-a0ff-91531fbfc58a","Type":"ContainerStarted","Data":"d90bfb4aafa1bedf9e4b41bf66c588dcbd87e5d79f735884816740b98c1e7c5b"}
Sep 11 21:39:20.886522 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:20.886481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp" event={"ID":"94fd6c68-511d-43f8-a0ff-91531fbfc58a","Type":"ContainerStarted","Data":"f9eae56c0a4883786e391cfa8c8bff272be1ff916764c877e733d6e41862b81f"}
Sep 11 21:39:20.886933 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:20.886616    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:20.902463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:20.902410    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp" podStartSLOduration=1.6458734339999999 podStartE2EDuration="3.902395562s" totalImagesPullingTime="2.256522128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:39:20.901594384 +0000 UTC m=+526.864619035" watchObservedRunningTime="2026-09-11 21:39:20.902395562 +0000 UTC m=+526.865420221"
Sep 11 21:39:31.893500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:31.893419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bplxp"
Sep 11 21:39:37.007160 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:37.007127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:37.109087 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:37.109058    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:37.513879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:37.513826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:47.145672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:47.145634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:47.303132 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:47.303093    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:49.426034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:49.426002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:54.511955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:54.511913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:54.611933 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:54.611882    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:54.701512 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:54.701477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:54.783456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:54.783338    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:55.363217 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:55.363187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:57.995781 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:57.995749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:39:58.095480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:39:58.095448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:30.611444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:30.611408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:30.703094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:30.703056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.008345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.008307    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.108822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.108790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.360108 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.360020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.421074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.421040    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.702010 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.701923    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.746272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.746237    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:40:32.794933 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.794892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:32.896334 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.896304    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"]
Sep 11 21:40:32.899703 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.899685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:32.901949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.901927    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 21:40:32.901949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.901939    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bc798\""
Sep 11 21:40:32.910264 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.910242    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"]
Sep 11 21:40:32.992958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.992922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2klf5\" (UniqueName: \"kubernetes.io/projected/98656f8b-93e9-4f6c-972e-9823cf5b908a-kube-api-access-2klf5\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:32.992958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:32.992959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656f8b-93e9-4f6c-972e-9823cf5b908a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.093975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.093925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2klf5\" (UniqueName: \"kubernetes.io/projected/98656f8b-93e9-4f6c-972e-9823cf5b908a-kube-api-access-2klf5\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.093975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.093976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656f8b-93e9-4f6c-972e-9823cf5b908a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.094748 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.094703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98656f8b-93e9-4f6c-972e-9823cf5b908a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.102249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.102224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2klf5\" (UniqueName: \"kubernetes.io/projected/98656f8b-93e9-4f6c-972e-9823cf5b908a-kube-api-access-2klf5\") pod \"payload-pre-processing-59fbc47bf8-8gmbg\" (UID: \"98656f8b-93e9-4f6c-972e-9823cf5b908a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.210304 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.210268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:33.483214 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:33.483183    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"]
Sep 11 21:40:33.486443 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:40:33.486415    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98656f8b_93e9_4f6c_972e_9823cf5b908a.slice/crio-84a480c39c53d75571266012fbfc5f0815345e8cd251fa81d3cf0526cf16406e WatchSource:0}: Error finding container 84a480c39c53d75571266012fbfc5f0815345e8cd251fa81d3cf0526cf16406e: Status 404 returned error can't find the container with id 84a480c39c53d75571266012fbfc5f0815345e8cd251fa81d3cf0526cf16406e
Sep 11 21:40:34.213308 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:34.213270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg" event={"ID":"98656f8b-93e9-4f6c-972e-9823cf5b908a","Type":"ContainerStarted","Data":"84a480c39c53d75571266012fbfc5f0815345e8cd251fa81d3cf0526cf16406e"}
Sep 11 21:40:35.220656 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:35.220558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg" event={"ID":"98656f8b-93e9-4f6c-972e-9823cf5b908a","Type":"ContainerStarted","Data":"60cc2bf59025144f8abba33bf88489b4d42c476de29db89eabf5256c2c573246"}
Sep 11 21:40:35.220656 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:35.220648    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:35.237075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:35.237019    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg" podStartSLOduration=1.822776248 podStartE2EDuration="3.237002727s" totalImagesPullingTime="1.414226479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:40:35.236059805 +0000 UTC m=+601.199084466" watchObservedRunningTime="2026-09-11 21:40:35.237002727 +0000 UTC m=+601.200027367"
Sep 11 21:40:43.199695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:43.199648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:43.261493 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:43.261461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:46.227603 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:46.227571    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-8gmbg"
Sep 11 21:40:54.908870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:54.908831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:40:55.461874 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:40:55.461831    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:41:09.884922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:09.884885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:11.105418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:11.105384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:11.989730 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:11.989677    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:12.662077 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:12.662040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:41:23.962555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:23.962521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:24.856385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:24.856350    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:41:30.979154 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:30.979118    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:32.159857 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:32.159818    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:41:49.602666 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:49.602625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:51.168328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:51.168291    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:41:59.861855 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.861816    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"]
Sep 11 21:41:59.865007 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.864983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.867770 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.867743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 21:41:59.867925 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.867776    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-442fp\""
Sep 11 21:41:59.867925 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.867792    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 21:41:59.867925 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.867792    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 21:41:59.873651 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.873628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"]
Sep 11 21:41:59.887454 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.887422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:41:59.943113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.943113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.943302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq52f\" (UniqueName: \"kubernetes.io/projected/d477bf29-b6dc-48b7-86b1-b804a0f338dc-kube-api-access-sq52f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.943302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.943302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:41:59.943400 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:41:59.943346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044301 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq52f\" (UniqueName: \"kubernetes.io/projected/d477bf29-b6dc-48b7-86b1-b804a0f338dc-kube-api-access-sq52f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044749    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.044920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.044878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.046907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.046882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d477bf29-b6dc-48b7-86b1-b804a0f338dc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.047055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.047036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d477bf29-b6dc-48b7-86b1-b804a0f338dc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.052566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.052546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq52f\" (UniqueName: \"kubernetes.io/projected/d477bf29-b6dc-48b7-86b1-b804a0f338dc-kube-api-access-sq52f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndcbh\" (UID: \"d477bf29-b6dc-48b7-86b1-b804a0f338dc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.176849 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.176764    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:00.446701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.446675    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"]
Sep 11 21:42:00.449257 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:42:00.449226    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd477bf29_b6dc_48b7_86b1_b804a0f338dc.slice/crio-35c2a1bda91ba0de415f108c565f044112ffcd882b5837e24bdc1774086edbb8 WatchSource:0}: Error finding container 35c2a1bda91ba0de415f108c565f044112ffcd882b5837e24bdc1774086edbb8: Status 404 returned error can't find the container with id 35c2a1bda91ba0de415f108c565f044112ffcd882b5837e24bdc1774086edbb8
Sep 11 21:42:00.450550 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.450530    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:42:00.603304 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:00.603268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh" event={"ID":"d477bf29-b6dc-48b7-86b1-b804a0f338dc","Type":"ContainerStarted","Data":"35c2a1bda91ba0de415f108c565f044112ffcd882b5837e24bdc1774086edbb8"}
Sep 11 21:42:01.609786 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:01.609752    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh" event={"ID":"d477bf29-b6dc-48b7-86b1-b804a0f338dc","Type":"ContainerStarted","Data":"8bbb6dd3775c9e239aa20b91655cfd51c56e14914a57a75a37f9182eb5a88f7c"}
Sep 11 21:42:01.629775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:01.629709    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh" podStartSLOduration=1.666252045 podStartE2EDuration="2.629692871s" totalImagesPullingTime="963.440826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:42:01.62868446 +0000 UTC m=+687.591709112" watchObservedRunningTime="2026-09-11 21:42:01.629692871 +0000 UTC m=+687.592717520"
Sep 11 21:42:01.961160 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:01.961122    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:42:04.282162 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:04.282132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:42:09.774435 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:09.774400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:42:10.177783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:10.177682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:10.177940 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:10.177791    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:10.190328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:10.190297    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:42:10.661664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:42:10.661634    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndcbh"
Sep 11 21:43:23.261516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:23.261482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:23.404298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:23.404272    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:24.396099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:24.396062    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:24.487325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:24.487282    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:39.595909 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:39.595865    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:39.657584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:39.657540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:39.728471 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:39.728432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:39.936725 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:39.936614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:40.013818 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:40.013784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:40.093556 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:40.093506    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:40.193382 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:40.193293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:40.294084 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:40.294054    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:40.394278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:40.394245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:41.043304 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:41.043259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:41.133512 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:41.133477    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:41.553154 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:41.553113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:42.817710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:42.817668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:42.913558 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:42.913531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:43.024486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.024454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:43.121672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.121586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:43.287150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.287119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:43.385390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.385296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:43.883809 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.883775    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:43:43.983598 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.983566    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:43:43.986075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.986055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:43.988314 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.988294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 21:43:43.988314 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.988310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e919c5-serving-cert\""
Sep 11 21:43:43.988916 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.988898    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 21:43:43.989068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.989046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2kttz\""
Sep 11 21:43:43.989068 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.989058    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:43:43.997269 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:43.997159    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:43:44.035054 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.035015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:44.078192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.078155    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pt87\" (UniqueName: \"kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.078192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.078196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.078400 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.078221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.078400 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.078361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.133091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.133050    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:44.178865 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.178786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.178865 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.178829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pt87\" (UniqueName: \"kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.178865 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.178855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.179101 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.178877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.179101 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:43:44.178925    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-serving-cert: secret "maas-api-e2e-inf-a-e919c5-serving-cert" not found
Sep 11 21:43:44.179101 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:43:44.178991    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls podName:c9a17eb6-8480-439d-a313-9fcd0bd24c33 nodeName:}" failed. No retries permitted until 2026-09-11 21:43:44.678976519 +0000 UTC m=+790.642001150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls") pod "maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" (UID: "c9a17eb6-8480-439d-a313-9fcd0bd24c33") : secret "maas-api-e2e-inf-a-e919c5-serving-cert" not found
Sep 11 21:43:44.179589 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.179571    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.181549 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.181533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.186730 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.186697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pt87\" (UniqueName: \"kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.274820 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.274775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:44.522421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.522388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:44.619144 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.619106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:44.622143 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.622115    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:43:44.624471 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.624453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.626746 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.626709    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e919c5\""
Sep 11 21:43:44.638287 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.638262    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:43:44.664571 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.664538    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:43:44.667389 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.667363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.669995 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.669965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:43:44.670955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.670913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e919c5-dockercfg-r7ks8\""
Sep 11 21:43:44.683971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.683943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.686566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.686539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.785459 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.785359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7tz\" (UniqueName: \"kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.785459 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.785444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs8p\" (UniqueName: \"kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.785672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.785509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.785672 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.785547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.886168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.886138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7tz\" (UniqueName: \"kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.886653 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.886189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs8p\" (UniqueName: \"kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.886653 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.886234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.886653 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.886256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.887028 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.887010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.887090 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.887042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.893835 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.893808    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7tz\" (UniqueName: \"kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz\") pod \"payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:44.893835 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.893824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs8p\" (UniqueName: \"kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p\") pod \"payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.898654 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.898629    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:44.935233 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.935197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:44.982880 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:44.982845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:45.179224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.179195    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:43:45.181752 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:45.181701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a17eb6_8480_439d_a313_9fcd0bd24c33.slice/crio-eb90ec4fc62f77a98429ba96cabd18974c1d6a04e6c99c20831ab4a97b1a40b8 WatchSource:0}: Error finding container eb90ec4fc62f77a98429ba96cabd18974c1d6a04e6c99c20831ab4a97b1a40b8: Status 404 returned error can't find the container with id eb90ec4fc62f77a98429ba96cabd18974c1d6a04e6c99c20831ab4a97b1a40b8
Sep 11 21:43:45.187508 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.187482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:45.215249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.215184    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:43:45.217563 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:45.217538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9380e66_d701_4874_b0e9_a139941233ea.slice/crio-b1477e05e21b217a23ca11f373e5ff112dc2b15d19c625a1abd1596e7c3a7039 WatchSource:0}: Error finding container b1477e05e21b217a23ca11f373e5ff112dc2b15d19c625a1abd1596e7c3a7039: Status 404 returned error can't find the container with id b1477e05e21b217a23ca11f373e5ff112dc2b15d19c625a1abd1596e7c3a7039
Sep 11 21:43:45.273077 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.272962    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:43:45.275801 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:45.275767    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea32560_d251_4109_9b55_a9a6387e4ed2.slice/crio-2050d777fb3c98b91be2e6424e3f674e0f3bbce9c049d96ffd85b3aaefb167c4 WatchSource:0}: Error finding container 2050d777fb3c98b91be2e6424e3f674e0f3bbce9c049d96ffd85b3aaefb167c4: Status 404 returned error can't find the container with id 2050d777fb3c98b91be2e6424e3f674e0f3bbce9c049d96ffd85b3aaefb167c4
Sep 11 21:43:45.276704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.276668    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:45.338770 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.338738    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:45.747441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.747409    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:45.844195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.844164    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:43:45.844499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.844466    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:45.846701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.846677    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:45.848697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.848671    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-35e777\""
Sep 11 21:43:45.848846 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.848775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-35e777-dockercfg-j6zwq\""
Sep 11 21:43:45.855529 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.855501    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:43:45.890380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.890337    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:43:45.893119 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.893095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:45.906251 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.906221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:43:45.998895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.998758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n2jv\" (UniqueName: \"kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:45.998895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.998836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:45.998895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.998864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wbh\" (UniqueName: \"kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:45.998895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:45.998894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.090574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.090535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" event={"ID":"c9380e66-d701-4874-b0e9-a139941233ea","Type":"ContainerStarted","Data":"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"}
Sep 11 21:43:46.090793 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.090604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" event={"ID":"c9380e66-d701-4874-b0e9-a139941233ea","Type":"ContainerStarted","Data":"b1477e05e21b217a23ca11f373e5ff112dc2b15d19c625a1abd1596e7c3a7039"}
Sep 11 21:43:46.091450 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.091414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:46.093191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.093162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" event={"ID":"c9a17eb6-8480-439d-a313-9fcd0bd24c33","Type":"ContainerStarted","Data":"eb90ec4fc62f77a98429ba96cabd18974c1d6a04e6c99c20831ab4a97b1a40b8"}
Sep 11 21:43:46.096150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.096100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" event={"ID":"8ea32560-d251-4109-9b55-a9a6387e4ed2","Type":"ContainerStarted","Data":"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"}
Sep 11 21:43:46.096150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.096149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" event={"ID":"8ea32560-d251-4109-9b55-a9a6387e4ed2","Type":"ContainerStarted","Data":"2050d777fb3c98b91be2e6424e3f674e0f3bbce9c049d96ffd85b3aaefb167c4"}
Sep 11 21:43:46.096329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.096258    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:46.099471 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.099440    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n2jv\" (UniqueName: \"kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.099605 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.099503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:46.099605 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.099531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wbh\" (UniqueName: \"kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:46.099605 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.099559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.100443 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.100417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.100575 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.100552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:46.109898 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.109785    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" podStartSLOduration=2.109761811 podStartE2EDuration="2.109761811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:46.108249631 +0000 UTC m=+792.071274280" watchObservedRunningTime="2026-09-11 21:43:46.109761811 +0000 UTC m=+792.072786462"
Sep 11 21:43:46.113738 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.113652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n2jv\" (UniqueName: \"kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv\") pod \"payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.116197 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.116150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wbh\" (UniqueName: \"kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh\") pod \"payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:46.129058 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.128990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" podStartSLOduration=2.128969576 podStartE2EDuration="2.128969576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:46.127042552 +0000 UTC m=+792.090067202" watchObservedRunningTime="2026-09-11 21:43:46.128969576 +0000 UTC m=+792.091994224"
Sep 11 21:43:46.158479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.158441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:46.205662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.205625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:46.210807 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.210774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:46.286156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.285652    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:46.448944 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.448906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:46.464409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.464193    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:43:46.474421 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:46.474386    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b104980_a2df_41d5_8311_0156528795e5.slice/crio-4ebda89366d3d102b9c8a7ea1b339cd07cf3cfc7493575839dcedb547ab92ac1 WatchSource:0}: Error finding container 4ebda89366d3d102b9c8a7ea1b339cd07cf3cfc7493575839dcedb547ab92ac1: Status 404 returned error can't find the container with id 4ebda89366d3d102b9c8a7ea1b339cd07cf3cfc7493575839dcedb547ab92ac1
Sep 11 21:43:46.529474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.529435    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:43:46.533946 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:46.533909    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ccb7a2_5993_4285_ba13_ce281c34e6dd.slice/crio-c3ea9f37d125bbb3aac8432ffdd2a755e892697097bed8a63de8352fbe9bd076 WatchSource:0}: Error finding container c3ea9f37d125bbb3aac8432ffdd2a755e892697097bed8a63de8352fbe9bd076: Status 404 returned error can't find the container with id c3ea9f37d125bbb3aac8432ffdd2a755e892697097bed8a63de8352fbe9bd076
Sep 11 21:43:46.675607 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.675471    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:46.776568 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.776428    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:46.785239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.785198    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:43:46.788988 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.788912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:46.791958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.791932    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f12741-dockercfg-5v2t8\""
Sep 11 21:43:46.792469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.792448    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f12741\""
Sep 11 21:43:46.799397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.798675    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:43:46.819561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.819526    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:43:46.823498 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.823466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:46.834796 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.834763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:43:46.916424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.916125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:46.916424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.916186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:46.916424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.916215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xvz\" (UniqueName: \"kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:46.916424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:46.916279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvl6m\" (UniqueName: \"kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.018099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.017320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvl6m\" (UniqueName: \"kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.018099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.017456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:47.018099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.017495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.018099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.017524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xvz\" (UniqueName: \"kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:47.018707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.018678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:47.018854 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.018678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.026464 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.026436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xvz\" (UniqueName: \"kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz\") pod \"payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:47.026601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.026526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvl6m\" (UniqueName: \"kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m\") pod \"payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.107760 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.107247    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" event={"ID":"8b104980-a2df-41d5-8311-0156528795e5","Type":"ContainerStarted","Data":"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"}
Sep 11 21:43:47.107760 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.107733    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:47.107985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.107766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" event={"ID":"8b104980-a2df-41d5-8311-0156528795e5","Type":"ContainerStarted","Data":"4ebda89366d3d102b9c8a7ea1b339cd07cf3cfc7493575839dcedb547ab92ac1"}
Sep 11 21:43:47.109277 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.109246    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" event={"ID":"99ccb7a2-5993-4285-ba13-ce281c34e6dd","Type":"ContainerStarted","Data":"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"}
Sep 11 21:43:47.109412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.109299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" event={"ID":"99ccb7a2-5993-4285-ba13-ce281c34e6dd","Type":"ContainerStarted","Data":"c3ea9f37d125bbb3aac8432ffdd2a755e892697097bed8a63de8352fbe9bd076"}
Sep 11 21:43:47.114298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.114261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:47.125273 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.125214    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" podStartSLOduration=2.1251957360000002 podStartE2EDuration="2.125195736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:47.123382596 +0000 UTC m=+793.086407269" watchObservedRunningTime="2026-09-11 21:43:47.125195736 +0000 UTC m=+793.088220384"
Sep 11 21:43:47.139920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.139871    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" podStartSLOduration=2.139853834 podStartE2EDuration="2.139853834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:47.138775106 +0000 UTC m=+793.101799756" watchObservedRunningTime="2026-09-11 21:43:47.139853834 +0000 UTC m=+793.102878482"
Sep 11 21:43:47.147452 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.147417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:47.206773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:47.206701    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.087892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.087855    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.116055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.116024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:43:48.327139 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.327083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.394765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.394648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.487230 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.487197    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:43:48.490677 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:48.490629    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cf08aa_6bc2_4ac8_ad14_d5e449adc697.slice/crio-0a27833f08d3676b19168ab298f86ee632c6781a475f7da55c4fa2ec51079e56 WatchSource:0}: Error finding container 0a27833f08d3676b19168ab298f86ee632c6781a475f7da55c4fa2ec51079e56: Status 404 returned error can't find the container with id 0a27833f08d3676b19168ab298f86ee632c6781a475f7da55c4fa2ec51079e56
Sep 11 21:43:48.490795 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.490750    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:43:48.492758 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:48.492709    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9be77bd_b319_4984_a54d_f1021178f2ef.slice/crio-6df9fbaff79706cbf9a68cf4cb0e726835ed826e67c7318b2a643edf1b35a875 WatchSource:0}: Error finding container 6df9fbaff79706cbf9a68cf4cb0e726835ed826e67c7318b2a643edf1b35a875: Status 404 returned error can't find the container with id 6df9fbaff79706cbf9a68cf4cb0e726835ed826e67c7318b2a643edf1b35a875
Sep 11 21:43:48.549556 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.549523    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.651874 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.651790    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.876921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.876879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:48.978231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:48.978176    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:49.075149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.075116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:49.123081 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.123047    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" event={"ID":"a9be77bd-b319-4984-a54d-f1021178f2ef","Type":"ContainerStarted","Data":"6a76205b778fb697517097dd91527b8ceb860edc90b718afbb6440a7913a8483"}
Sep 11 21:43:49.123551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.123112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" event={"ID":"a9be77bd-b319-4984-a54d-f1021178f2ef","Type":"ContainerStarted","Data":"6df9fbaff79706cbf9a68cf4cb0e726835ed826e67c7318b2a643edf1b35a875"}
Sep 11 21:43:49.123551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.123202    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:43:49.124950 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.124915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" event={"ID":"90cf08aa-6bc2-4ac8-ad14-d5e449adc697","Type":"ContainerStarted","Data":"3967c8b43df84d85501e32406f595b773582dbd053038f13b63c30dc6b2af45e"}
Sep 11 21:43:49.125085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.124959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" event={"ID":"90cf08aa-6bc2-4ac8-ad14-d5e449adc697","Type":"ContainerStarted","Data":"0a27833f08d3676b19168ab298f86ee632c6781a475f7da55c4fa2ec51079e56"}
Sep 11 21:43:49.125085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.125056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:43:49.126688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.126666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" event={"ID":"c9a17eb6-8480-439d-a313-9fcd0bd24c33","Type":"ContainerStarted","Data":"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"}
Sep 11 21:43:49.126829 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.126814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:49.143049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.142747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" podStartSLOduration=3.142708867 podStartE2EDuration="3.142708867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:49.142652738 +0000 UTC m=+795.105677391" watchObservedRunningTime="2026-09-11 21:43:49.142708867 +0000 UTC m=+795.105733536"
Sep 11 21:43:49.160936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.160881    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" podStartSLOduration=3.160863177 podStartE2EDuration="3.160863177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:49.15899406 +0000 UTC m=+795.122018710" watchObservedRunningTime="2026-09-11 21:43:49.160863177 +0000 UTC m=+795.123887826"
Sep 11 21:43:49.177078 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.177016    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" podStartSLOduration=3.112698276 podStartE2EDuration="6.17699644s" totalImagesPullingTime="3.064298164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:49.175409839 +0000 UTC m=+795.138434491" watchObservedRunningTime="2026-09-11 21:43:49.17699644 +0000 UTC m=+795.140021090"
Sep 11 21:43:49.222470 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:49.222420    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:50.464862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:50.464825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:51.162066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:51.162030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:51.275985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:51.275948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:51.448950 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:51.448851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:51.826995 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:51.826952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:51.910742 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:51.910680    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:52.526312 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:52.526263    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:53.409827 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:53.409789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.206123 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.206081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.309421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.309385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.410386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.410347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.781011 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.780975    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.875863 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.875826    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:43:54.878070 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.878044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:54.878408 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.878391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:54.880482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.880462    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5b160c\""
Sep 11 21:43:54.888786 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.888761    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:43:54.924291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.924258    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:43:54.926520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.926503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:54.928691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.928671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5b160c-dockercfg-rkl4q\""
Sep 11 21:43:54.937822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.937798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:43:54.991417 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.991382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:54.991635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:54.991445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.045626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.045540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:55.092497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.092454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.092497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.092502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bk5h\" (UniqueName: \"kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.092710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.092540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.092710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.092608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.093244 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.093226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.102219 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.102192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5\") pod \"payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.137538 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.137502    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:43:55.190331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.190298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:55.193960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.193934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bk5h\" (UniqueName: \"kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.194118 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.193988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.195032 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.195003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.203255 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.203232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bk5h\" (UniqueName: \"kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h\") pod \"payload-processing-e2e-autores-5b160c-6d47696cf-prcbn\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") " pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.237190 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.237156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:55.358049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.358020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:55.463281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.463247    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:55.464689 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.464641    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:43:55.467018 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:55.466992    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1949585a_cffe_45d1_8d96_ce3d2d82972a.slice/crio-e983beef5f5561341840ac30512d7baee597e8d1a48f5add2b7058dde8f06c26 WatchSource:0}: Error finding container e983beef5f5561341840ac30512d7baee597e8d1a48f5add2b7058dde8f06c26: Status 404 returned error can't find the container with id e983beef5f5561341840ac30512d7baee597e8d1a48f5add2b7058dde8f06c26
Sep 11 21:43:55.533745 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.533687    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:43:55.535681 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:55.535653    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767cbaaa_6472_4bc8_aeff_8244cd03becf.slice/crio-b498a801fd21c931a338cc020078a36cd8bb630da74e7ef3800838bfa169b9c8 WatchSource:0}: Error finding container b498a801fd21c931a338cc020078a36cd8bb630da74e7ef3800838bfa169b9c8: Status 404 returned error can't find the container with id b498a801fd21c931a338cc020078a36cd8bb630da74e7ef3800838bfa169b9c8
Sep 11 21:43:55.948044 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:55.947951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:56.048431 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.048397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:56.056344 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.056307    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:43:56.058800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.058778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.060927 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.060906    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b2d1bd14\""
Sep 11 21:43:56.061056 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.061023    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b2d1bd14-dockercfg-lqthq\""
Sep 11 21:43:56.072612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.072585    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:43:56.089329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.089293    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:43:56.091780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.091763    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.104405 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.104368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:43:56.163876 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.163843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" event={"ID":"767cbaaa-6472-4bc8-aeff-8244cd03becf","Type":"ContainerStarted","Data":"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"}
Sep 11 21:43:56.164052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.163903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" event={"ID":"767cbaaa-6472-4bc8-aeff-8244cd03becf","Type":"ContainerStarted","Data":"b498a801fd21c931a338cc020078a36cd8bb630da74e7ef3800838bfa169b9c8"}
Sep 11 21:43:56.164052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.163939    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:43:56.165544 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.165517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" event={"ID":"1949585a-cffe-45d1-8d96-ce3d2d82972a","Type":"ContainerStarted","Data":"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"}
Sep 11 21:43:56.165662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.165564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" event={"ID":"1949585a-cffe-45d1-8d96-ce3d2d82972a","Type":"ContainerStarted","Data":"e983beef5f5561341840ac30512d7baee597e8d1a48f5add2b7058dde8f06c26"}
Sep 11 21:43:56.165662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.165587    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:43:56.182626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.182579    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" podStartSLOduration=2.182561453 podStartE2EDuration="2.182561453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:56.180657689 +0000 UTC m=+802.143682339" watchObservedRunningTime="2026-09-11 21:43:56.182561453 +0000 UTC m=+802.145586105"
Sep 11 21:43:56.207289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.203329    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" podStartSLOduration=2.203308165 podStartE2EDuration="2.203308165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:56.195236561 +0000 UTC m=+802.158261212" watchObservedRunningTime="2026-09-11 21:43:56.203308165 +0000 UTC m=+802.166332817"
Sep 11 21:43:56.207289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.205023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.207289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.205086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrpx\" (UniqueName: \"kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.207289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.205175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jj9m\" (UniqueName: \"kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.207289 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.205234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.306538 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.306497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.306747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.306547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrpx\" (UniqueName: \"kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.306747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.306600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jj9m\" (UniqueName: \"kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.306747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.306646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.307360 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.307335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.307572 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.307552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.314588 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.314560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jj9m\" (UniqueName: \"kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m\") pod \"payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") " pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.314808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.314744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrpx\" (UniqueName: \"kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx\") pod \"payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.368962 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.368931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:56.409361 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.409326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:56.523948 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.523889    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:56.646573 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.646548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:43:56.648673 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:56.648647    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ef672e_0228_4f32_9b8c_40d959adec52.slice/crio-0a47a6115077f6f4ce49a55aab271c0a9b20f0614b7de91ee07a34bbc1d46d9b WatchSource:0}: Error finding container 0a47a6115077f6f4ce49a55aab271c0a9b20f0614b7de91ee07a34bbc1d46d9b: Status 404 returned error can't find the container with id 0a47a6115077f6f4ce49a55aab271c0a9b20f0614b7de91ee07a34bbc1d46d9b
Sep 11 21:43:56.700527 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:56.700491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:43:56.703542 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:43:56.703505    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0bfa4f_57c3_4577_848f_830c2785fbd4.slice/crio-d4aef7f9aaa9689560ef49b741dc388c30f6ecbd402ca6da047dba8643719ff8 WatchSource:0}: Error finding container d4aef7f9aaa9689560ef49b741dc388c30f6ecbd402ca6da047dba8643719ff8: Status 404 returned error can't find the container with id d4aef7f9aaa9689560ef49b741dc388c30f6ecbd402ca6da047dba8643719ff8
Sep 11 21:43:57.111620 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.111523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:43:57.173796 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.173758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" event={"ID":"d1ef672e-0228-4f32-9b8c-40d959adec52","Type":"ContainerStarted","Data":"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"}
Sep 11 21:43:57.173958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.173821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" event={"ID":"d1ef672e-0228-4f32-9b8c-40d959adec52","Type":"ContainerStarted","Data":"0a47a6115077f6f4ce49a55aab271c0a9b20f0614b7de91ee07a34bbc1d46d9b"}
Sep 11 21:43:57.173958 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.173918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:43:57.175624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.175595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" event={"ID":"4b0bfa4f-57c3-4577-848f-830c2785fbd4","Type":"ContainerStarted","Data":"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"}
Sep 11 21:43:57.175864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.175847    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:43:57.175930 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.175882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" event={"ID":"4b0bfa4f-57c3-4577-848f-830c2785fbd4","Type":"ContainerStarted","Data":"d4aef7f9aaa9689560ef49b741dc388c30f6ecbd402ca6da047dba8643719ff8"}
Sep 11 21:43:57.198636 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.198576    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" podStartSLOduration=1.198555107 podStartE2EDuration="1.198555107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:57.196915708 +0000 UTC m=+803.159940358" watchObservedRunningTime="2026-09-11 21:43:57.198555107 +0000 UTC m=+803.161579762"
Sep 11 21:43:57.214840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.214783    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" podStartSLOduration=1.2147636529999999 podStartE2EDuration="1.214763653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:43:57.213286068 +0000 UTC m=+803.176310720" watchObservedRunningTime="2026-09-11 21:43:57.214763653 +0000 UTC m=+803.177788302"
Sep 11 21:43:57.561441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:57.561405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:58.118313 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:58.118281    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:43:58.118515 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:58.118349    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:43:58.502150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:58.502103    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:43:59.128591 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:43:59.128558    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:44:00.132924 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.132894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:44:00.133385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.133235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:44:00.252178 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.252113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:00.417495 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.417406    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:00.514122 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.514086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:00.614368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:00.614325    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:01.211621 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:01.211583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:03.348496 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:03.348460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:04.270298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:04.270267    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:44:04.919451 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:04.919416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:05.029768 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:05.029710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:06.537830 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:06.537784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:07.178141 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:07.178110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:44:07.178336 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:07.178191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:44:07.514463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:07.514417    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:07.694404 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:07.694363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:07.798060 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:07.797969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:08.034978 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.034944    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:08.181972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.181887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:44:08.182125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.182008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:44:08.287333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.287298    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:08.407578 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.407542    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:08.565584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.565549    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:08.656526 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:08.656474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:09.700069 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:09.700028    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:10.387577 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:10.387540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:10.492085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:10.492056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:11.357862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:11.357831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:11.566806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:11.566758    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:12.017345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:12.017314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:12.168553 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:12.168519    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:12.421619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:12.421528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:13.262397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:13.262357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:13.365615 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:13.365583    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:13.769520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:13.769480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:13.870622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:13.870586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:13.970398 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:13.970362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:14.071301 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:14.071198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:14.884707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:14.884671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:15.701267 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:15.701233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:15.802286 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:15.802227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:16.153076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.153038    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:16.355284 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.355251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:16.451219 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.451124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:16.490616 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.490588    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:44:16.494015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.493993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.496285 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.496259    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7e6f5e-dockercfg-5fctf\""
Sep 11 21:44:16.496414 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.496259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7e6f5e\""
Sep 11 21:44:16.503846 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.503823    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:44:16.590298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.590271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94jd9\" (UniqueName: \"kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.590298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.590303    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.691907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.691870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94jd9\" (UniqueName: \"kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.691907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.691911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.692970 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.692937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.700302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.700277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94jd9\" (UniqueName: \"kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9\") pod \"payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:16.795865 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.795826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:16.806637 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:16.806604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:17.080013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.079985    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:44:17.082412 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:44:17.082386    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de58ae2_5cc9_4fbe_8350_5e7fb8b41547.slice/crio-d60b03c1a8ca6f10ca1d15c6e37bb625aba77bec56eaf410be320bc230279ac5 WatchSource:0}: Error finding container d60b03c1a8ca6f10ca1d15c6e37bb625aba77bec56eaf410be320bc230279ac5: Status 404 returned error can't find the container with id d60b03c1a8ca6f10ca1d15c6e37bb625aba77bec56eaf410be320bc230279ac5
Sep 11 21:44:17.283789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.283743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" event={"ID":"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547","Type":"ContainerStarted","Data":"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"}
Sep 11 21:44:17.284213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.283829    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:17.284213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.283859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" event={"ID":"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547","Type":"ContainerStarted","Data":"d60b03c1a8ca6f10ca1d15c6e37bb625aba77bec56eaf410be320bc230279ac5"}
Sep 11 21:44:17.302053 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.301986    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" podStartSLOduration=1.301965509 podStartE2EDuration="1.301965509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:17.300853391 +0000 UTC m=+823.263878041" watchObservedRunningTime="2026-09-11 21:44:17.301965509 +0000 UTC m=+823.264990162"
Sep 11 21:44:17.709281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:17.709243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:18.702655 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:18.702617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:19.405300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:19.405258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:19.875406 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:19.875365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:19.980471 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:19.980436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:20.091789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:20.091751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:20.306975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:20.306939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:21.716619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:21.716568    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:21.866691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:21.866651    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:44:22.601229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.601196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:22.636374 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.636342    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:44:22.636640 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.636605    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" podUID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" containerName="payload-processing" containerID="cri-o://12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498" gracePeriod=30
Sep 11 21:44:22.640565 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.640114    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:44:22.640565 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.640320    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" podUID="d1ef672e-0228-4f32-9b8c-40d959adec52" containerName="payload-pre-processing" containerID="cri-o://614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19" gracePeriod=30
Sep 11 21:44:22.680445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.680418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:22.747332 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.747293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:22.816359 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.816326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:22.876271 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.876193    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:22.912287 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.912262    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:44:22.915864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.915845    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:44:22.971693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:22.971655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:23.053505 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.053468    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume\") pod \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") "
Sep 11 21:44:23.053691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.053523    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jj9m\" (UniqueName: \"kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m\") pod \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\" (UID: \"4b0bfa4f-57c3-4577-848f-830c2785fbd4\") "
Sep 11 21:44:23.053691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.053553    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume\") pod \"d1ef672e-0228-4f32-9b8c-40d959adec52\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") "
Sep 11 21:44:23.053691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.053660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lrpx\" (UniqueName: \"kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx\") pod \"d1ef672e-0228-4f32-9b8c-40d959adec52\" (UID: \"d1ef672e-0228-4f32-9b8c-40d959adec52\") "
Sep 11 21:44:23.054168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.054004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1ef672e-0228-4f32-9b8c-40d959adec52" (UID: "d1ef672e-0228-4f32-9b8c-40d959adec52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:23.054168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.054009    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b0bfa4f-57c3-4577-848f-830c2785fbd4" (UID: "4b0bfa4f-57c3-4577-848f-830c2785fbd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:23.054243 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.054202    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0bfa4f-57c3-4577-848f-830c2785fbd4-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:23.054243 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.054224    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ef672e-0228-4f32-9b8c-40d959adec52-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:23.056074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.056051    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m" (OuterVolumeSpecName: "kube-api-access-7jj9m") pod "4b0bfa4f-57c3-4577-848f-830c2785fbd4" (UID: "4b0bfa4f-57c3-4577-848f-830c2785fbd4"). InnerVolumeSpecName "kube-api-access-7jj9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:23.056236 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.056216    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx" (OuterVolumeSpecName: "kube-api-access-7lrpx") pod "d1ef672e-0228-4f32-9b8c-40d959adec52" (UID: "d1ef672e-0228-4f32-9b8c-40d959adec52"). InnerVolumeSpecName "kube-api-access-7lrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:23.155643 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.155554    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jj9m\" (UniqueName: \"kubernetes.io/projected/4b0bfa4f-57c3-4577-848f-830c2785fbd4-kube-api-access-7jj9m\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:23.155643 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.155584    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrpx\" (UniqueName: \"kubernetes.io/projected/d1ef672e-0228-4f32-9b8c-40d959adec52-kube-api-access-7lrpx\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:23.299219 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.299180    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:23.316832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.316806    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"
Sep 11 21:44:23.317016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.316803    2582 generic.go:356] "Generic (PLEG): container finished" podID="d1ef672e-0228-4f32-9b8c-40d959adec52" containerID="614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19" exitCode=0
Sep 11 21:44:23.317016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.316886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" event={"ID":"d1ef672e-0228-4f32-9b8c-40d959adec52","Type":"ContainerDied","Data":"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"}
Sep 11 21:44:23.317016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.316956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92" event={"ID":"d1ef672e-0228-4f32-9b8c-40d959adec52","Type":"ContainerDied","Data":"0a47a6115077f6f4ce49a55aab271c0a9b20f0614b7de91ee07a34bbc1d46d9b"}
Sep 11 21:44:23.317016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.317005    2582 scope.go:117] "RemoveContainer" containerID="614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"
Sep 11 21:44:23.318827 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.318700    2582 generic.go:356] "Generic (PLEG): container finished" podID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" containerID="12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498" exitCode=0
Sep 11 21:44:23.318827 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.318765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" event={"ID":"4b0bfa4f-57c3-4577-848f-830c2785fbd4","Type":"ContainerDied","Data":"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"}
Sep 11 21:44:23.318827 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.318782    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"
Sep 11 21:44:23.319031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.318983    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl" event={"ID":"4b0bfa4f-57c3-4577-848f-830c2785fbd4","Type":"ContainerDied","Data":"d4aef7f9aaa9689560ef49b741dc388c30f6ecbd402ca6da047dba8643719ff8"}
Sep 11 21:44:23.327595 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.327572    2582 scope.go:117] "RemoveContainer" containerID="614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"
Sep 11 21:44:23.327922 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:44:23.327894    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19\": container with ID starting with 614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19 not found: ID does not exist" containerID="614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"
Sep 11 21:44:23.327984 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.327935    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19"} err="failed to get container status \"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19\": rpc error: code = NotFound desc = could not find container \"614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19\": container with ID starting with 614c21dcc4cbbb6bfe421bf8d64cd0888c8d461d4b82dd822aa5b028b2190f19 not found: ID does not exist"
Sep 11 21:44:23.327984 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.327957    2582 scope.go:117] "RemoveContainer" containerID="12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"
Sep 11 21:44:23.336603 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.336587    2582 scope.go:117] "RemoveContainer" containerID="12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"
Sep 11 21:44:23.336946 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:44:23.336929    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498\": container with ID starting with 12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498 not found: ID does not exist" containerID="12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"
Sep 11 21:44:23.337007 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.336953    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498"} err="failed to get container status \"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498\": rpc error: code = NotFound desc = could not find container \"12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498\": container with ID starting with 12e226408da9d1f23adf12b2f0de6c50790d10e1ef9efa463414af6cbc3f9498 not found: ID does not exist"
Sep 11 21:44:23.340228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.340207    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:44:23.344130 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.344108    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2d1bd14-77cb5bd6d6-66b92"]
Sep 11 21:44:23.354118 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.354093    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:44:23.362743 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.359003    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b2d1bd14-5df598bc9-p7wdl"]
Sep 11 21:44:23.424773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.424674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:23.524276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.524236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:23.645924 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.645890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:23.979393 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:23.979359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:24.185074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:24.185034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:24.293115 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:24.293030    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:24.454845 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:24.454805    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" path="/var/lib/kubelet/pods/4b0bfa4f-57c3-4577-848f-830c2785fbd4/volumes"
Sep 11 21:44:24.455240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:24.455221    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ef672e-0228-4f32-9b8c-40d959adec52" path="/var/lib/kubelet/pods/d1ef672e-0228-4f32-9b8c-40d959adec52/volumes"
Sep 11 21:44:25.168480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:25.168447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:25.991417 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:25.991377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:26.526165 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:26.526137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:26.645459 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:26.645426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:26.728822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:26.728780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:26.830384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:26.830306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:26.928884 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:26.928847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:27.895917 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:27.895881    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:28.290767 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:28.290705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:44:28.733940 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:28.733909    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:28.831432 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:28.831399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:29.061490 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.061393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:29.385997 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.385906    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:29.495174 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.495139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:29.558986 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.558952    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:29.559397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559384    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" containerName="payload-processing"
Sep 11 21:44:29.559441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559399    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" containerName="payload-processing"
Sep 11 21:44:29.559441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559412    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ef672e-0228-4f32-9b8c-40d959adec52" containerName="payload-pre-processing"
Sep 11 21:44:29.559441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559419    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ef672e-0228-4f32-9b8c-40d959adec52" containerName="payload-pre-processing"
Sep 11 21:44:29.559557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559478    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0bfa4f-57c3-4577-848f-830c2785fbd4" containerName="payload-processing"
Sep 11 21:44:29.559557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.559487    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ef672e-0228-4f32-9b8c-40d959adec52" containerName="payload-pre-processing"
Sep 11 21:44:29.562466 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.562448    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.564946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.564925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-762b41ed\""
Sep 11 21:44:29.565052 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.564928    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-762b41ed-dockercfg-snpgt\""
Sep 11 21:44:29.573472 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.573447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:29.718739 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.718621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvz82\" (UniqueName: \"kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.718914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.718783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.819918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.819879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.820100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.819954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvz82\" (UniqueName: \"kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.820594 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.820574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.824450 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.824416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:29.828334 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.828309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvz82\" (UniqueName: \"kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82\") pod \"payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:29.873690 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:29.873643    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:30.147351 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.147256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:30.150051 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:44:30.150011    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ad7d0_c0fb_4bc5_b24d_ddc0224d828c.slice/crio-cdf064bf83bcbaa96515dbc2e44d1264026fa9ad6cc80f3006c8d7fc1192c86b WatchSource:0}: Error finding container cdf064bf83bcbaa96515dbc2e44d1264026fa9ad6cc80f3006c8d7fc1192c86b: Status 404 returned error can't find the container with id cdf064bf83bcbaa96515dbc2e44d1264026fa9ad6cc80f3006c8d7fc1192c86b
Sep 11 21:44:30.359205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.359172    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" event={"ID":"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c","Type":"ContainerStarted","Data":"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"}
Sep 11 21:44:30.359205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.359224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" event={"ID":"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c","Type":"ContainerStarted","Data":"cdf064bf83bcbaa96515dbc2e44d1264026fa9ad6cc80f3006c8d7fc1192c86b"}
Sep 11 21:44:30.359462 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.359273    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:30.375537 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.375486    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" podStartSLOduration=1.375470473 podStartE2EDuration="1.375470473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:44:30.373589451 +0000 UTC m=+836.336614113" watchObservedRunningTime="2026-09-11 21:44:30.375470473 +0000 UTC m=+836.338495165"
Sep 11 21:44:30.588855 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:30.588824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:31.517688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:31.517648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:31.736681 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:31.736648    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:32.168765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:32.168703    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:32.930586 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:32.930543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:33.030627 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:33.030596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:33.107814 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:33.107780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:33.782414 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:33.782377    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:34.536214 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:34.536177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:36.995708 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:36.995675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:38.357095 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:38.357057    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:44:41.366754 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:41.366702    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:47.368839 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:47.368798    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:47.429296 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:47.429260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:49.385245 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:49.385202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:49.485679 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:49.485647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:49.804369 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:49.804336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:51.173215 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:51.173161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:51.919804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:51.919766    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:52.769710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:52.769672    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:53.394410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:53.394373    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:54.025362 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:54.025321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:54.746178 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:54.746140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:55.386959 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:55.386929    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:56.200907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:56.200871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:57.786943 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:57.786902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.412278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.412241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.447709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.447676    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:58.447992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.447964    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" podUID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" containerName="payload-pre-processing" containerID="cri-o://9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25" gracePeriod=30
Sep 11 21:44:58.513489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.513446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.620864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.620765    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.712003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.711979    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:58.724488 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.724452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.783678 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.783644    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvz82\" (UniqueName: \"kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82\") pod \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") "
Sep 11 21:44:58.783889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.783698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume\") pod \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\" (UID: \"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c\") "
Sep 11 21:44:58.784108 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.784084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" (UID: "f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:44:58.786166 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.786129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82" (OuterVolumeSpecName: "kube-api-access-qvz82") pod "f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" (UID: "f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c"). InnerVolumeSpecName "kube-api-access-qvz82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:44:58.819211 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.819163    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:58.885028 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.884964    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvz82\" (UniqueName: \"kubernetes.io/projected/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-kube-api-access-qvz82\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:58.885028 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:58.884999    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:44:59.124982 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.124940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:59.252932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.252892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:44:59.510156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.510119    2582 generic.go:356] "Generic (PLEG): container finished" podID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" containerID="9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25" exitCode=0
Sep 11 21:44:59.510156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.510162    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"
Sep 11 21:44:59.510420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.510187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" event={"ID":"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c","Type":"ContainerDied","Data":"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"}
Sep 11 21:44:59.510420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.510241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg" event={"ID":"f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c","Type":"ContainerDied","Data":"cdf064bf83bcbaa96515dbc2e44d1264026fa9ad6cc80f3006c8d7fc1192c86b"}
Sep 11 21:44:59.510420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.510269    2582 scope.go:117] "RemoveContainer" containerID="9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"
Sep 11 21:44:59.519797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.519779    2582 scope.go:117] "RemoveContainer" containerID="9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"
Sep 11 21:44:59.520122 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:44:59.520103    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25\": container with ID starting with 9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25 not found: ID does not exist" containerID="9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"
Sep 11 21:44:59.520177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.520130    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25"} err="failed to get container status \"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25\": rpc error: code = NotFound desc = could not find container \"9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25\": container with ID starting with 9df1e7ee8de40057b9c7b57a671e02323e51ddbb12925b7110f7d6fcb5948b25 not found: ID does not exist"
Sep 11 21:44:59.540692 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.540656    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:59.548641 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.548610    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-762b41ed-9cd4d855d-s8dpg"]
Sep 11 21:44:59.597732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:44:59.597674    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:00.095050 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.095015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:00.147329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.147294    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"]
Sep 11 21:45:00.147762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.147749    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" containerName="payload-pre-processing"
Sep 11 21:45:00.147762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.147763    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" containerName="payload-pre-processing"
Sep 11 21:45:00.147844 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.147826    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" containerName="payload-pre-processing"
Sep 11 21:45:00.152295 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.152264    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"]
Sep 11 21:45:00.152474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.152399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"
Sep 11 21:45:00.154638 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.154610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:45:00.158313 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.157454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"
Sep 11 21:45:00.166413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.166332    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"]
Sep 11 21:45:00.170263 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.170241    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"
Sep 11 21:45:00.170547 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.170519    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"]
Sep 11 21:45:00.182570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.181748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"
Sep 11 21:45:00.252162 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.252111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:00.456456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.456341    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c" path="/var/lib/kubelet/pods/f33ad7d0-c0fb-4bc5-b24d-ddc0224d828c/volumes"
Sep 11 21:45:00.457324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.457296    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"]
Sep 11 21:45:00.459708 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:45:00.459681    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39faff82_975f_4614_8731_4fe3d411b374.slice/crio-8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda WatchSource:0}: Error finding container 8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda: Status 404 returned error can't find the container with id 8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda
Sep 11 21:45:00.474331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.474304    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"]
Sep 11 21:45:00.476175 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:45:00.476145    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa07be3_8db2_4346_9b8c_01bd49506205.slice/crio-f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde WatchSource:0}: Error finding container f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde: Status 404 returned error can't find the container with id f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde
Sep 11 21:45:00.517703 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.517668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk" event={"ID":"7fa07be3-8db2-4346-9b8c-01bd49506205","Type":"ContainerStarted","Data":"f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde"}
Sep 11 21:45:00.518961 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.518933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq" event={"ID":"39faff82-975f-4614-8731-4fe3d411b374","Type":"ContainerStarted","Data":"8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda"}
Sep 11 21:45:00.779221 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:00.779182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:01.584673 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:01.584634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:02.417335 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:02.417293    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:02.642934 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:02.642884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:02.729833 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:02.729794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:03.202323 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:03.202290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:03.538664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:03.538627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk" event={"ID":"7fa07be3-8db2-4346-9b8c-01bd49506205","Type":"ContainerStarted","Data":"843c285ecee3306d914f63e36eff89ac1f35c875c86cef902090a412c94bc769"}
Sep 11 21:45:03.540311 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:03.540282    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq" event={"ID":"39faff82-975f-4614-8731-4fe3d411b374","Type":"ContainerStarted","Data":"bec4c68c57436db115611b95c7af0a1717bd22c36198f03c708c05fb6c89fb0c"}
Sep 11 21:45:03.552739 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:03.552606    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk" podStartSLOduration=1.051929153 podStartE2EDuration="3.552589507s" totalImagesPullingTime="2.500660354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:03.550232466 +0000 UTC m=+869.513257148" watchObservedRunningTime="2026-09-11 21:45:03.552589507 +0000 UTC m=+869.515614215"
Sep 11 21:45:03.563977 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:03.563931    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq" podStartSLOduration=1.050887055 podStartE2EDuration="3.563915263s" totalImagesPullingTime="2.513028208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:03.562780222 +0000 UTC m=+869.525804872" watchObservedRunningTime="2026-09-11 21:45:03.563915263 +0000 UTC m=+869.526939911"
Sep 11 21:45:04.016571 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.016529    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:04.120273 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.120240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:04.335180 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.335147    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:04.547324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.547287    2582 generic.go:356] "Generic (PLEG): container finished" podID="7fa07be3-8db2-4346-9b8c-01bd49506205" containerID="843c285ecee3306d914f63e36eff89ac1f35c875c86cef902090a412c94bc769" exitCode=0
Sep 11 21:45:04.547506 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.547380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk" event={"ID":"7fa07be3-8db2-4346-9b8c-01bd49506205","Type":"ContainerDied","Data":"843c285ecee3306d914f63e36eff89ac1f35c875c86cef902090a412c94bc769"}
Sep 11 21:45:04.548915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.548894    2582 generic.go:356] "Generic (PLEG): container finished" podID="39faff82-975f-4614-8731-4fe3d411b374" containerID="bec4c68c57436db115611b95c7af0a1717bd22c36198f03c708c05fb6c89fb0c" exitCode=0
Sep 11 21:45:04.549030 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.548975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq" event={"ID":"39faff82-975f-4614-8731-4fe3d411b374","Type":"ContainerDied","Data":"bec4c68c57436db115611b95c7af0a1717bd22c36198f03c708c05fb6c89fb0c"}
Sep 11 21:45:04.615576 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.615484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:04.714976 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.714939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:04.869310 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:04.869227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:45:05.118372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:05.118336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:05.713176 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:05.713151    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"
Sep 11 21:45:05.736961 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:05.736939    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"
Sep 11 21:45:05.815473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:05.815383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:06.487259 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.487226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:06.561709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.561685    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"
Sep 11 21:45:06.561709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.561687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk" event={"ID":"7fa07be3-8db2-4346-9b8c-01bd49506205","Type":"ContainerDied","Data":"f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde"}
Sep 11 21:45:06.561963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.561805    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e0c54d4d5eb857a63483cb8154ef6ddd9a8021ff6526ad3215b649f94eabde"
Sep 11 21:45:06.563013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.562992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq" event={"ID":"39faff82-975f-4614-8731-4fe3d411b374","Type":"ContainerDied","Data":"8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda"}
Sep 11 21:45:06.563125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.563016    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"
Sep 11 21:45:06.563125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:06.563033    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8777006f0498dbd90209714dfb9160c0b67d861c91bb300a26f8e61cc9f4ceda"
Sep 11 21:45:07.326596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:07.326566    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:07.958236 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:07.958196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:08.107096 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:08.107060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:20.371433 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:20.371394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:45:25.365671 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:25.365637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:26.190213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:26.190124    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:26.834297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:26.834260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:27.504069 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:27.504041    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:28.295327 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:28.295288    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:29.106976 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:29.106934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:29.759077 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:29.759044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:30.491605 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:30.491575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:31.149293 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:31.149259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:31.804705 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:31.804670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:32.598401 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:32.598363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:32.670774 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:32.670740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:45:33.407474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:33.407433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:34.200903 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:34.200864    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:34.807859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:34.807772    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:34.882882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:34.882846    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:34.957139 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:34.957094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:35.035191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:35.035149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:35.107420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:35.107320    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:35.482013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:35.481980    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:36.183866 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:36.183826    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:36.284048 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:36.284010    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:36.623751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:36.623695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:36.929098 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:36.929017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:37.029013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:37.028940    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:37.128628 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:37.128592    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:38.091307 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:38.091261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:38.897425 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:38.897399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:39.522456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:39.522413    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:39.623001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:39.622951    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:39.873586 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:39.873497    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:40.593880 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:40.593835    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:41.288040 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:41.288008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:42.098990 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:42.098955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:42.867433 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:42.867397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:43.470732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.470684    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471147    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39faff82-975f-4614-8731-4fe3d411b374" containerName="cleanup"
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471160    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39faff82-975f-4614-8731-4fe3d411b374" containerName="cleanup"
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471194    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa07be3-8db2-4346-9b8c-01bd49506205" containerName="cleanup"
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471201    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa07be3-8db2-4346-9b8c-01bd49506205" containerName="cleanup"
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471264    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39faff82-975f-4614-8731-4fe3d411b374" containerName="cleanup"
Sep 11 21:45:43.472971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.471275    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa07be3-8db2-4346-9b8c-01bd49506205" containerName="cleanup"
Sep 11 21:45:43.474002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.473982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.475932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.475915    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7a9aa317-serving-cert\""
Sep 11 21:45:43.483327 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.483279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:45:43.487538 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.487516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.487669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.487544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn6x\" (UniqueName: \"kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.487669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.487575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.487815 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.487706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.508094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.508055    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:43.588435 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.588396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.588628 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.588444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn6x\" (UniqueName: \"kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.588628 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.588582    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.588821 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.588773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.589248 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:45:43.588780    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-serving-cert: secret "maas-api-e2e-derive-7a9aa317-serving-cert" not found
Sep 11 21:45:43.589248 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:45:43.589010    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls podName:7b03eb6c-5918-4572-8855-cebf22cd4d78 nodeName:}" failed. No retries permitted until 2026-09-11 21:45:44.0889872 +0000 UTC m=+910.052011851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls") pod "maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" (UID: "7b03eb6c-5918-4572-8855-cebf22cd4d78") : secret "maas-api-e2e-derive-7a9aa317-serving-cert" not found
Sep 11 21:45:43.590032 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.590007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.592403 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.592381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.599046 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.599023    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn6x\" (UniqueName: \"kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:43.608458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.608435    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:43.772186 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.772145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:43.952373 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:43.952337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:44.053465 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.053369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:45:44.094841 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.094795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:44.097474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.097443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") pod \"maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:44.387370 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.387275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:44.664014 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.663984    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:45:44.666371 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:45:44.666343    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b03eb6c_5918_4572_8855_cebf22cd4d78.slice/crio-dcc1dab05fe0f32dc1b5deec4b55d9edece05cc05a830723c0ef199cd3912cbc WatchSource:0}: Error finding container dcc1dab05fe0f32dc1b5deec4b55d9edece05cc05a830723c0ef199cd3912cbc: Status 404 returned error can't find the container with id dcc1dab05fe0f32dc1b5deec4b55d9edece05cc05a830723c0ef199cd3912cbc
Sep 11 21:45:44.770373 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:44.770339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" event={"ID":"7b03eb6c-5918-4572-8855-cebf22cd4d78","Type":"ContainerStarted","Data":"dcc1dab05fe0f32dc1b5deec4b55d9edece05cc05a830723c0ef199cd3912cbc"}
Sep 11 21:45:45.777237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:45.777202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" event={"ID":"7b03eb6c-5918-4572-8855-cebf22cd4d78","Type":"ContainerStarted","Data":"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"}
Sep 11 21:45:45.777623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:45.777306    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:45.796684 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:45.796635    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" podStartSLOduration=2.395481295 podStartE2EDuration="2.796612306s" totalImagesPullingTime="401.131011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:45:45.794134733 +0000 UTC m=+911.757159384" watchObservedRunningTime="2026-09-11 21:45:45.796612306 +0000 UTC m=+911.759636955"
Sep 11 21:45:51.788962 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:51.788928    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:45:52.266188 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:52.266148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:45:55.839554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:45:55.839521    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:00.254102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:00.254070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:01.064526 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:01.064489    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:01.917932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:01.917892    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:02.602275 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:02.602237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:03.371427 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:03.371389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:03.658280 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:03.658197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:04.209202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:04.209170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:05.068786 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:05.068741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:06.100801 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:06.100768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:06.921378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:06.921345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:07.705202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:07.705165    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:08.536993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:08.536960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:09.491791 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:09.491755    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:09.881518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:09.881433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:10.231822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:10.231784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:11.184271 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.184236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:11.671293 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.671259    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:46:11.790566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.790534    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"]
Sep 11 21:46:11.798507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.798481    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5b160c-29819385-xqflq"]
Sep 11 21:46:11.870172 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.870139    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:46:11.870422 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.870390    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" podUID="767cbaaa-6472-4bc8-aeff-8244cd03becf" containerName="payload-processing" containerID="cri-o://9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a" gracePeriod=30
Sep 11 21:46:11.879284 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.879260    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:46:11.879487 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:11.879466    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" podUID="1949585a-cffe-45d1-8d96-ce3d2d82972a" containerName="payload-pre-processing" containerID="cri-o://9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde" gracePeriod=30
Sep 11 21:46:12.169571 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.169547    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:46:12.172669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.172649    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:46:12.257188 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume\") pod \"767cbaaa-6472-4bc8-aeff-8244cd03becf\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") "
Sep 11 21:46:12.257610 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume\") pod \"1949585a-cffe-45d1-8d96-ce3d2d82972a\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") "
Sep 11 21:46:12.257610 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257314    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bk5h\" (UniqueName: \"kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h\") pod \"767cbaaa-6472-4bc8-aeff-8244cd03becf\" (UID: \"767cbaaa-6472-4bc8-aeff-8244cd03becf\") "
Sep 11 21:46:12.257610 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5\") pod \"1949585a-cffe-45d1-8d96-ce3d2d82972a\" (UID: \"1949585a-cffe-45d1-8d96-ce3d2d82972a\") "
Sep 11 21:46:12.257808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "767cbaaa-6472-4bc8-aeff-8244cd03becf" (UID: "767cbaaa-6472-4bc8-aeff-8244cd03becf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:12.257808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.257642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1949585a-cffe-45d1-8d96-ce3d2d82972a" (UID: "1949585a-cffe-45d1-8d96-ce3d2d82972a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:12.259614 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.259582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5" (OuterVolumeSpecName: "kube-api-access-wtwm5") pod "1949585a-cffe-45d1-8d96-ce3d2d82972a" (UID: "1949585a-cffe-45d1-8d96-ce3d2d82972a"). InnerVolumeSpecName "kube-api-access-wtwm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:12.259763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.259587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h" (OuterVolumeSpecName: "kube-api-access-5bk5h") pod "767cbaaa-6472-4bc8-aeff-8244cd03becf" (UID: "767cbaaa-6472-4bc8-aeff-8244cd03becf"). InnerVolumeSpecName "kube-api-access-5bk5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:12.358183 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.358146    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1949585a-cffe-45d1-8d96-ce3d2d82972a-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:12.358183 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.358173    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bk5h\" (UniqueName: \"kubernetes.io/projected/767cbaaa-6472-4bc8-aeff-8244cd03becf-kube-api-access-5bk5h\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:12.358183 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.358183    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwm5\" (UniqueName: \"kubernetes.io/projected/1949585a-cffe-45d1-8d96-ce3d2d82972a-kube-api-access-wtwm5\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:12.358183 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.358192    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767cbaaa-6472-4bc8-aeff-8244cd03becf-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:12.461775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.461742    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39faff82-975f-4614-8731-4fe3d411b374" path="/var/lib/kubelet/pods/39faff82-975f-4614-8731-4fe3d411b374/volumes"
Sep 11 21:46:12.499748 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.499682    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:12.918578 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.918537    2582 generic.go:356] "Generic (PLEG): container finished" podID="767cbaaa-6472-4bc8-aeff-8244cd03becf" containerID="9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a" exitCode=0
Sep 11 21:46:12.918800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.918587    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"
Sep 11 21:46:12.918800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.918613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" event={"ID":"767cbaaa-6472-4bc8-aeff-8244cd03becf","Type":"ContainerDied","Data":"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"}
Sep 11 21:46:12.918800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.918663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn" event={"ID":"767cbaaa-6472-4bc8-aeff-8244cd03becf","Type":"ContainerDied","Data":"b498a801fd21c931a338cc020078a36cd8bb630da74e7ef3800838bfa169b9c8"}
Sep 11 21:46:12.918800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.918687    2582 scope.go:117] "RemoveContainer" containerID="9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"
Sep 11 21:46:12.920431 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.920359    2582 generic.go:356] "Generic (PLEG): container finished" podID="1949585a-cffe-45d1-8d96-ce3d2d82972a" containerID="9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde" exitCode=0
Sep 11 21:46:12.920431 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.920425    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" event={"ID":"1949585a-cffe-45d1-8d96-ce3d2d82972a","Type":"ContainerDied","Data":"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"}
Sep 11 21:46:12.920602 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.920442    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"
Sep 11 21:46:12.920602 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.920472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq" event={"ID":"1949585a-cffe-45d1-8d96-ce3d2d82972a","Type":"ContainerDied","Data":"e983beef5f5561341840ac30512d7baee597e8d1a48f5add2b7058dde8f06c26"}
Sep 11 21:46:12.929141 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.929118    2582 scope.go:117] "RemoveContainer" containerID="9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"
Sep 11 21:46:12.929516 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:46:12.929491    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a\": container with ID starting with 9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a not found: ID does not exist" containerID="9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"
Sep 11 21:46:12.929630 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.929527    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a"} err="failed to get container status \"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a\": rpc error: code = NotFound desc = could not find container \"9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a\": container with ID starting with 9a4b82f1fc2be1c66e6425b7f3ac6400587f599eca00a2cb0948463089ee1d3a not found: ID does not exist"
Sep 11 21:46:12.929630 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.929549    2582 scope.go:117] "RemoveContainer" containerID="9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"
Sep 11 21:46:12.938015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.937983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:12.938167 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.938122    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:46:12.939759 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.939736    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5b160c-6d47696cf-prcbn"]
Sep 11 21:46:12.941463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.941444    2582 scope.go:117] "RemoveContainer" containerID="9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"
Sep 11 21:46:12.941812 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:46:12.941792    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde\": container with ID starting with 9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde not found: ID does not exist" containerID="9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"
Sep 11 21:46:12.941893 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.941822    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde"} err="failed to get container status \"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde\": rpc error: code = NotFound desc = could not find container \"9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde\": container with ID starting with 9eaf528b65bda9f54dde27de74f94318331b8b55b0595270421b417602da6cde not found: ID does not exist"
Sep 11 21:46:12.951419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.951371    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:46:12.953578 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:12.953554    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5b160c-69796f8978-bwpbq"]
Sep 11 21:46:13.870881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:13.870845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:14.454842 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:14.454807    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1949585a-cffe-45d1-8d96-ce3d2d82972a" path="/var/lib/kubelet/pods/1949585a-cffe-45d1-8d96-ce3d2d82972a/volumes"
Sep 11 21:46:14.455231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:14.455218    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767cbaaa-6472-4bc8-aeff-8244cd03becf" path="/var/lib/kubelet/pods/767cbaaa-6472-4bc8-aeff-8244cd03becf/volumes"
Sep 11 21:46:14.522945 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:14.522915    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:15.186241 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:15.186210    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:15.890018 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:15.889981    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:16.530157 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:16.530121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:17.508386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:17.508350    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.299180 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.299144    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:46:21.299943 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.299669    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" podUID="7b03eb6c-5918-4572-8855-cebf22cd4d78" containerName="maas-api" containerID="cri-o://3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178" gracePeriod=30
Sep 11 21:46:21.331708 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.331675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.411921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.411875    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.481387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.481352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.553504 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.553442    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:46:21.574860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.574822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.680530 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.680491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:21.752485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.752445    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljn6x\" (UniqueName: \"kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x\") pod \"7b03eb6c-5918-4572-8855-cebf22cd4d78\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") "
Sep 11 21:46:21.752665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.752519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") pod \"7b03eb6c-5918-4572-8855-cebf22cd4d78\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") "
Sep 11 21:46:21.752665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.752577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs\") pod \"7b03eb6c-5918-4572-8855-cebf22cd4d78\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") "
Sep 11 21:46:21.752665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.752603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca\") pod \"7b03eb6c-5918-4572-8855-cebf22cd4d78\" (UID: \"7b03eb6c-5918-4572-8855-cebf22cd4d78\") "
Sep 11 21:46:21.753278 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.753207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b03eb6c-5918-4572-8855-cebf22cd4d78" (UID: "7b03eb6c-5918-4572-8855-cebf22cd4d78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:46:21.755204 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.755179    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b03eb6c-5918-4572-8855-cebf22cd4d78" (UID: "7b03eb6c-5918-4572-8855-cebf22cd4d78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:46:21.758482 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.755339    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x" (OuterVolumeSpecName: "kube-api-access-ljn6x") pod "7b03eb6c-5918-4572-8855-cebf22cd4d78" (UID: "7b03eb6c-5918-4572-8855-cebf22cd4d78"). InnerVolumeSpecName "kube-api-access-ljn6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:46:21.759463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.759378    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b03eb6c-5918-4572-8855-cebf22cd4d78" (UID: "7b03eb6c-5918-4572-8855-cebf22cd4d78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:46:21.854413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.854311    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn6x\" (UniqueName: \"kubernetes.io/projected/7b03eb6c-5918-4572-8855-cebf22cd4d78-kube-api-access-ljn6x\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.854413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.854346    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.854413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.854358    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b03eb6c-5918-4572-8855-cebf22cd4d78-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.854413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.854369    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b03eb6c-5918-4572-8855-cebf22cd4d78-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:46:21.973182 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.973144    2582 generic.go:356] "Generic (PLEG): container finished" podID="7b03eb6c-5918-4572-8855-cebf22cd4d78" containerID="3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178" exitCode=0
Sep 11 21:46:21.973375 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.973195    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"
Sep 11 21:46:21.973375 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.973240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" event={"ID":"7b03eb6c-5918-4572-8855-cebf22cd4d78","Type":"ContainerDied","Data":"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"}
Sep 11 21:46:21.973375 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.973297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl" event={"ID":"7b03eb6c-5918-4572-8855-cebf22cd4d78","Type":"ContainerDied","Data":"dcc1dab05fe0f32dc1b5deec4b55d9edece05cc05a830723c0ef199cd3912cbc"}
Sep 11 21:46:21.973375 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.973325    2582 scope.go:117] "RemoveContainer" containerID="3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"
Sep 11 21:46:21.986467 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.986443    2582 scope.go:117] "RemoveContainer" containerID="3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"
Sep 11 21:46:21.986831 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:46:21.986806    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178\": container with ID starting with 3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178 not found: ID does not exist" containerID="3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"
Sep 11 21:46:21.986919 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.986841    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178"} err="failed to get container status \"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178\": rpc error: code = NotFound desc = could not find container \"3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178\": container with ID starting with 3b1b16e8e0eab6faad6597867ed3713e86e566eb8038360f241d69d221b37178 not found: ID does not exist"
Sep 11 21:46:21.997855 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.997825    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:46:21.999345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:21.999321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:22.000025 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.000005    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7a9aa317-769c6ccd48-5h7hl"]
Sep 11 21:46:22.380930 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.380891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:22.455171 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.455130    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b03eb6c-5918-4572-8855-cebf22cd4d78" path="/var/lib/kubelet/pods/7b03eb6c-5918-4572-8855-cebf22cd4d78/volumes"
Sep 11 21:46:22.480804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.480775    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:22.580832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.580796    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:22.789570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:22.789508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:23.608408 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:23.608369    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:23.917292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:23.917200    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:24.267461 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:24.267433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:24.986968 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:24.986934    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:28.271956 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:28.271914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:46:29.885455 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:29.885424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:29.949568 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:29.949533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:30.024533 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:30.024491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:30.124189 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:30.124153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:30.222836 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:30.222747    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:30.573125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:30.573092    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:36.260256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:36.260222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:54.325910 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:54.325854    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:54.595221 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:54.595132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:55.058410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:55.058376    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:55.362935 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:55.362850    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:55.468055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:55.468019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:56.505168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:56.505132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:57.702879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:57.702841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:58.048955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:58.048921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:58.632233 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:58.632190    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:58.724260 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:58.724222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:58.828861 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:58.828825    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:46:59.811929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:46:59.811886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:00.492797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:00.492757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:01.279240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:01.279209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.047270 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047241    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:47:02.047653 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047640    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767cbaaa-6472-4bc8-aeff-8244cd03becf" containerName="payload-processing"
Sep 11 21:47:02.047697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047655    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="767cbaaa-6472-4bc8-aeff-8244cd03becf" containerName="payload-processing"
Sep 11 21:47:02.047697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b03eb6c-5918-4572-8855-cebf22cd4d78" containerName="maas-api"
Sep 11 21:47:02.047697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047676    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b03eb6c-5918-4572-8855-cebf22cd4d78" containerName="maas-api"
Sep 11 21:47:02.047697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047696    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1949585a-cffe-45d1-8d96-ce3d2d82972a" containerName="payload-pre-processing"
Sep 11 21:47:02.047860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047701    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1949585a-cffe-45d1-8d96-ce3d2d82972a" containerName="payload-pre-processing"
Sep 11 21:47:02.047860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047774    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b03eb6c-5918-4572-8855-cebf22cd4d78" containerName="maas-api"
Sep 11 21:47:02.047860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1949585a-cffe-45d1-8d96-ce3d2d82972a" containerName="payload-pre-processing"
Sep 11 21:47:02.047860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.047790    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="767cbaaa-6472-4bc8-aeff-8244cd03becf" containerName="payload-processing"
Sep 11 21:47:02.050962 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.050939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.054036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.054020    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b75330-serving-cert\""
Sep 11 21:47:02.068100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.068063    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:47:02.096141 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.096089    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.192435 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.192403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.219970 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.219943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4zm\" (UniqueName: \"kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.220134 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.219988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.220134 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.220006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.220134 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.220024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.320772 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.320678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4zm\" (UniqueName: \"kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.320772 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.320741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.321268 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.320811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.321268 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:47:02.320845    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-serving-cert: secret "maas-api-e2e-explicit-b75330-serving-cert" not found
Sep 11 21:47:02.321268 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.320849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.321268 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:47:02.320923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls podName:c8acb32d-3cfb-4c29-bc5f-f790df07ee9d nodeName:}" failed. No retries permitted until 2026-09-11 21:47:02.8209015 +0000 UTC m=+988.783926128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls") pod "maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" (UID: "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d") : secret "maas-api-e2e-explicit-b75330-serving-cert" not found
Sep 11 21:47:02.321559 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.321541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.323633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.323614    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.329083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.329065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4zm\" (UniqueName: \"kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.448810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.448774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.556037 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.555999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.650454 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.650362    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:02.692122 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.692092    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:47:02.695888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.695864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.698074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.698050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b75330-dockercfg-nlp7t\""
Sep 11 21:47:02.698074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.698064    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b75330\""
Sep 11 21:47:02.706831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.706808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:47:02.825472 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.825435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.825702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.825489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.825702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.825570    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjl8\" (UniqueName: \"kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.828152 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.828127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") pod \"maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:02.926413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.926326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.926413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.926413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjl8\" (UniqueName: \"kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.927107 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.927082    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.934103 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.934081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjl8\" (UniqueName: \"kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8\") pod \"payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") " pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:02.961065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:02.961036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:03.007335 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.007299    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:03.232444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.232418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:47:03.234694 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:47:03.234659    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8acb32d_3cfb_4c29_bc5f_f790df07ee9d.slice/crio-8ad255d6e19bc2592a8845b60f8ee73e639c7f2783bca94e614bc8e0625da5b5 WatchSource:0}: Error finding container 8ad255d6e19bc2592a8845b60f8ee73e639c7f2783bca94e614bc8e0625da5b5: Status 404 returned error can't find the container with id 8ad255d6e19bc2592a8845b60f8ee73e639c7f2783bca94e614bc8e0625da5b5
Sep 11 21:47:03.235593 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.235484    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:03.236137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.236119    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:47:03.290299 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.290277    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:47:03.292594 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:47:03.292569    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2104f1a_0e01_43a5_a5c9_d342afd680dd.slice/crio-2aa3cd0bc4ffc28fc2bcf8814bce48aa42b8507f2dad959e8d9a318e1b5c7b7f WatchSource:0}: Error finding container 2aa3cd0bc4ffc28fc2bcf8814bce48aa42b8507f2dad959e8d9a318e1b5c7b7f: Status 404 returned error can't find the container with id 2aa3cd0bc4ffc28fc2bcf8814bce48aa42b8507f2dad959e8d9a318e1b5c7b7f
Sep 11 21:47:03.310830 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.310804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:03.409227 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:03.409182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:04.196169 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.196128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" event={"ID":"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d","Type":"ContainerStarted","Data":"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"}
Sep 11 21:47:04.196402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.196325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:04.196402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.196369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" event={"ID":"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d","Type":"ContainerStarted","Data":"8ad255d6e19bc2592a8845b60f8ee73e639c7f2783bca94e614bc8e0625da5b5"}
Sep 11 21:47:04.197708 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.197685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" event={"ID":"e2104f1a-0e01-43a5-a5c9-d342afd680dd","Type":"ContainerStarted","Data":"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"}
Sep 11 21:47:04.197837 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.197737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" event={"ID":"e2104f1a-0e01-43a5-a5c9-d342afd680dd","Type":"ContainerStarted","Data":"2aa3cd0bc4ffc28fc2bcf8814bce48aa42b8507f2dad959e8d9a318e1b5c7b7f"}
Sep 11 21:47:04.197837 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.197771    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:04.215354 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.215302    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" podStartSLOduration=1.797249122 podStartE2EDuration="2.215286728s" totalImagesPullingTime="418.037606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:04.212855844 +0000 UTC m=+990.175880493" watchObservedRunningTime="2026-09-11 21:47:04.215286728 +0000 UTC m=+990.178311408"
Sep 11 21:47:04.233384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:04.233334    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" podStartSLOduration=2.233319788 podStartE2EDuration="2.233319788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:47:04.231224017 +0000 UTC m=+990.194248679" watchObservedRunningTime="2026-09-11 21:47:04.233319788 +0000 UTC m=+990.196344454"
Sep 11 21:47:10.208213 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:10.208182    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:47:12.159801 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:12.159763    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:47:12.992899 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:12.992866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:15.205481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:15.205448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:47:19.341967 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:19.341935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:24.227216 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:24.227175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:25.060473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:25.060436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:25.641018 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:25.640989    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:25.748557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:25.748509    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:25.852241 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:25.852198    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:25.974587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:25.974491    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:26.741303 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:26.741274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:27.724512 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:27.724469    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:28.624303 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:28.624269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:28.724789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:28.724752    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:30.284631 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:30.284588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:32.057504 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:32.057468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:47:32.402439 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:32.402317    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:36.342789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:36.342748    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:37.127846 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:37.127785    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:42.087268 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:42.087238    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:43.216184 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:43.216141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:43.322294 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:43.322257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:43.418691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:43.418654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:43.516476 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:43.516438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:44.207166 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:44.207133    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:44.301647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:44.301610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:45.827295 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:45.827257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:51.364363 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:51.364328    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:47:52.706925 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:52.706887    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:52.791522 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:52.791483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:52.862202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:52.862162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:53.373114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:53.373083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:59.169938 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:59.169902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:59.270495 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:59.270456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:59.411048 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:59.411013    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:47:59.699848 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:47:59.699818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:00.348020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:00.347992    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:04.981102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:04.981069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:05.590429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:05.590385    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:05.685650 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:05.685611    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:05.787612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:05.787576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:06.592993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:06.592964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:07.365041 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:07.365001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:08.093614 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:08.093582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:08.897064 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:08.897034    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:09.678481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.678442    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:09.682270 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.682249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.684568 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.684546    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a21586a5-serving-cert\""
Sep 11 21:48:09.693229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.693201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:09.708858 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.708823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:09.752165 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.752131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.752337 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.752172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.752337 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.752214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmk79\" (UniqueName: \"kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.752337 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.752297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.770871 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.770838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:09.853291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.853249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.853291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.853298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.853519 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:48:09.853406    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-serving-cert: secret "maas-api-e2e-mt-a21586a5-serving-cert" not found
Sep 11 21:48:09.853519 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.853427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.853519 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:48:09.853492    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls podName:b27dce4b-051c-4fdd-bce7-92989be07ee0 nodeName:}" failed. No retries permitted until 2026-09-11 21:48:10.353472097 +0000 UTC m=+1056.316496726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls") pod "maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" (UID: "b27dce4b-051c-4fdd-bce7-92989be07ee0") : secret "maas-api-e2e-mt-a21586a5-serving-cert" not found
Sep 11 21:48:09.853657 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.853526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmk79\" (UniqueName: \"kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.854160 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.854140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.855811 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.855794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:09.861421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:09.861393    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmk79\" (UniqueName: \"kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:10.079065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.079029    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:10.272001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.271965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:10.359364 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.359244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:10.362075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.362046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") pod \"maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:10.367957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.367930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:10.406055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.406020    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:10.409976 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.409959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.412262 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.412241    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a21586a5\""
Sep 11 21:48:10.412443 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.412282    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a21586a5-dockercfg-rrm4z\""
Sep 11 21:48:10.419288 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.419263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:10.460588 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.460555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5tl\" (UniqueName: \"kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.460790 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.460773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.561475 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.561441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.561652 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.561494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5tl\" (UniqueName: \"kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.562205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.562186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.571753 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.571710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5tl\" (UniqueName: \"kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl\") pod \"payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") " pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.594600 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.594570    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:10.721889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.721851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:10.865896 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.865872    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:10.867899 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:48:10.867873    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27dce4b_051c_4fdd_bce7_92989be07ee0.slice/crio-048b788b9cca0e62dec60bcc4cbafe4bbb6c9cc4d93fa79e162fc1a6d1a08138 WatchSource:0}: Error finding container 048b788b9cca0e62dec60bcc4cbafe4bbb6c9cc4d93fa79e162fc1a6d1a08138: Status 404 returned error can't find the container with id 048b788b9cca0e62dec60bcc4cbafe4bbb6c9cc4d93fa79e162fc1a6d1a08138
Sep 11 21:48:10.994895 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:10.994870    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:10.997098 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:48:10.997069    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ab844c_a9b2_435b_8bae_ee8e24813f12.slice/crio-bd1891925081caba6fd7a8389c2c25c5fd8741023f8621da62053491a407eb21 WatchSource:0}: Error finding container bd1891925081caba6fd7a8389c2c25c5fd8741023f8621da62053491a407eb21: Status 404 returned error can't find the container with id bd1891925081caba6fd7a8389c2c25c5fd8741023f8621da62053491a407eb21
Sep 11 21:48:11.544771 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.544645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" event={"ID":"59ab844c-a9b2-435b-8bae-ee8e24813f12","Type":"ContainerStarted","Data":"59a5f536979e5970d21897f1dab69ee472a33c18a93b2218fe8c18e1484ddbf9"}
Sep 11 21:48:11.544771 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.544707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" event={"ID":"59ab844c-a9b2-435b-8bae-ee8e24813f12","Type":"ContainerStarted","Data":"bd1891925081caba6fd7a8389c2c25c5fd8741023f8621da62053491a407eb21"}
Sep 11 21:48:11.545030 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.544845    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:11.546400 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.546376    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" event={"ID":"b27dce4b-051c-4fdd-bce7-92989be07ee0","Type":"ContainerStarted","Data":"00a1c2085c063003a7e1b5094973bbe824995795a0b72d2e08dc13bcb6fb0c7a"}
Sep 11 21:48:11.546509 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.546411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" event={"ID":"b27dce4b-051c-4fdd-bce7-92989be07ee0","Type":"ContainerStarted","Data":"048b788b9cca0e62dec60bcc4cbafe4bbb6c9cc4d93fa79e162fc1a6d1a08138"}
Sep 11 21:48:11.546509 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.546430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:11.560731 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.560663    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" podStartSLOduration=1.560649324 podStartE2EDuration="1.560649324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:48:11.559948457 +0000 UTC m=+1057.522973108" watchObservedRunningTime="2026-09-11 21:48:11.560649324 +0000 UTC m=+1057.523673982"
Sep 11 21:48:11.576517 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:11.576464    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" podStartSLOduration=2.194435462 podStartE2EDuration="2.576450782s" totalImagesPullingTime="382.01532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:48:11.575036094 +0000 UTC m=+1057.538060743" watchObservedRunningTime="2026-09-11 21:48:11.576450782 +0000 UTC m=+1057.539475479"
Sep 11 21:48:17.557660 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:17.557627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:22.554307 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:22.554277    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:25.177151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:25.177115    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:48:25.612787 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:25.612675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:26.556323 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:26.556289    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:27.393022 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:27.392983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:28.101243 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:28.101207    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:28.833662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:28.833631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:29.719996 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:29.719955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:30.100523 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:30.100488    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:30.200816 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:30.200773    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:30.527061 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:30.527008    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:31.519081 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:31.519043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:32.213345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:32.213305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:33.125630 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:33.125596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:33.803369 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:33.803321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:34.649254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:34.649216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:35.329175 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:35.329140    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:35.664559 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:35.664456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:35.976543 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:35.976502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:36.300418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.300331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:36.404897 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.404857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:36.777340 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.777296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:36.838095 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.837000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:48:36.838095 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.837232    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" podUID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" containerName="maas-api" containerID="cri-o://9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749" gracePeriod=30
Sep 11 21:48:36.869333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.869302    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:36.900281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.900247    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:48:36.900461 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.900436    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" podUID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" containerName="payload-processing" containerID="cri-o://a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc" gracePeriod=30
Sep 11 21:48:36.967331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:36.967301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:37.064498 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.064412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:37.123240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.123201    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:37.166003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.165978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:48:37.209432 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.209405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:48:37.215413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca\") pod \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") "
Sep 11 21:48:37.215557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq4zm\" (UniqueName: \"kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm\") pod \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") "
Sep 11 21:48:37.215557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215479    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wjl8\" (UniqueName: \"kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8\") pod \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") "
Sep 11 21:48:37.215557 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume\") pod \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\" (UID: \"e2104f1a-0e01-43a5-a5c9-d342afd680dd\") "
Sep 11 21:48:37.215741 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") pod \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") "
Sep 11 21:48:37.215741 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215615    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs\") pod \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\" (UID: \"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d\") "
Sep 11 21:48:37.215840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" (UID: "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:37.215955 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.215937    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.216371 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.216158    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2104f1a-0e01-43a5-a5c9-d342afd680dd" (UID: "e2104f1a-0e01-43a5-a5c9-d342afd680dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:37.217888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.217832    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" (UID: "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:37.217972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.217914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" (UID: "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:37.218144 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.218127    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm" (OuterVolumeSpecName: "kube-api-access-mq4zm") pod "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" (UID: "c8acb32d-3cfb-4c29-bc5f-f790df07ee9d"). InnerVolumeSpecName "kube-api-access-mq4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:37.218192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.218152    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8" (OuterVolumeSpecName: "kube-api-access-9wjl8") pod "e2104f1a-0e01-43a5-a5c9-d342afd680dd" (UID: "e2104f1a-0e01-43a5-a5c9-d342afd680dd"). InnerVolumeSpecName "kube-api-access-9wjl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:37.317348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.317268    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4zm\" (UniqueName: \"kubernetes.io/projected/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-kube-api-access-mq4zm\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.317348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.317295    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjl8\" (UniqueName: \"kubernetes.io/projected/e2104f1a-0e01-43a5-a5c9-d342afd680dd-kube-api-access-9wjl8\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.317348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.317304    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2104f1a-0e01-43a5-a5c9-d342afd680dd-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.317348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.317315    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.317348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.317324    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:37.415954 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.415899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:37.517878 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.517849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:37.684633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.684543    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" containerID="a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc" exitCode=0
Sep 11 21:48:37.684633 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.684588    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"
Sep 11 21:48:37.684864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.684641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" event={"ID":"e2104f1a-0e01-43a5-a5c9-d342afd680dd","Type":"ContainerDied","Data":"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"}
Sep 11 21:48:37.684864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.684693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8" event={"ID":"e2104f1a-0e01-43a5-a5c9-d342afd680dd","Type":"ContainerDied","Data":"2aa3cd0bc4ffc28fc2bcf8814bce48aa42b8507f2dad959e8d9a318e1b5c7b7f"}
Sep 11 21:48:37.684864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.684750    2582 scope.go:117] "RemoveContainer" containerID="a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"
Sep 11 21:48:37.686165 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.686143    2582 generic.go:356] "Generic (PLEG): container finished" podID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" containerID="9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749" exitCode=0
Sep 11 21:48:37.686273 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.686188    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"
Sep 11 21:48:37.686329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.686187    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" event={"ID":"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d","Type":"ContainerDied","Data":"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"}
Sep 11 21:48:37.686388 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.686333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc" event={"ID":"c8acb32d-3cfb-4c29-bc5f-f790df07ee9d","Type":"ContainerDied","Data":"8ad255d6e19bc2592a8845b60f8ee73e639c7f2783bca94e614bc8e0625da5b5"}
Sep 11 21:48:37.694267 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.694231    2582 scope.go:117] "RemoveContainer" containerID="a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"
Sep 11 21:48:37.694658 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:48:37.694638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc\": container with ID starting with a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc not found: ID does not exist" containerID="a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"
Sep 11 21:48:37.694782 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.694666    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc"} err="failed to get container status \"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc\": rpc error: code = NotFound desc = could not find container \"a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc\": container with ID starting with a7c0a4d0ddcd1da55543a5534f25467ca696e3473fc654c963ecae993c280bfc not found: ID does not exist"
Sep 11 21:48:37.694782 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.694685    2582 scope.go:117] "RemoveContainer" containerID="9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"
Sep 11 21:48:37.704007 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.703989    2582 scope.go:117] "RemoveContainer" containerID="9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"
Sep 11 21:48:37.704266 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:48:37.704248    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749\": container with ID starting with 9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749 not found: ID does not exist" containerID="9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"
Sep 11 21:48:37.704335 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.704277    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749"} err="failed to get container status \"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749\": rpc error: code = NotFound desc = could not find container \"9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749\": container with ID starting with 9d7811be60373e3bdb07020c9080eb772521ace45fb5820eee18b63aba7ba749 not found: ID does not exist"
Sep 11 21:48:37.708628 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.708606    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:48:37.712293 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.712273    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b75330-5d46bcdfbf-mgkb8"]
Sep 11 21:48:37.721877 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.721856    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:48:37.726065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:37.726047    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b75330-c44f6f5d9-jkhnc"]
Sep 11 21:48:38.186504 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:38.186453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:38.456261 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:38.456172    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" path="/var/lib/kubelet/pods/c8acb32d-3cfb-4c29-bc5f-f790df07ee9d/volumes"
Sep 11 21:48:38.456558 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:38.456533    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" path="/var/lib/kubelet/pods/e2104f1a-0e01-43a5-a5c9-d342afd680dd/volumes"
Sep 11 21:48:38.918154 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:38.918122    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:39.779664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:39.779631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:39.905255 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:39.905219    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:40.629327 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:40.629290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:41.498205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:41.498149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:42.496961 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:42.496919    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:42.795019 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:42.794931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:42.864858 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:42.864823    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:43.381796 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:43.381761    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:44.203220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:44.203182    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:44.985004 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:44.984971    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:45.656787 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:45.656757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:46.552990 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:46.552950    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:47.332508 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:47.332467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:48.169968 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:48.169931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:48.970803 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:48.970768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:48:52.688491 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:52.688452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:53.590575 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.590347    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:53.590840 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.590648    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" podUID="b27dce4b-051c-4fdd-bce7-92989be07ee0" containerName="maas-api" containerID="cri-o://00a1c2085c063003a7e1b5094973bbe824995795a0b72d2e08dc13bcb6fb0c7a" gracePeriod=30
Sep 11 21:48:53.616107 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.616071    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:53.651610 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.651547    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:53.651904 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.651840    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" podUID="59ab844c-a9b2-435b-8bae-ee8e24813f12" containerName="payload-processing" containerID="cri-o://59a5f536979e5970d21897f1dab69ee472a33c18a93b2218fe8c18e1484ddbf9" gracePeriod=30
Sep 11 21:48:53.712632 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.712600    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:53.786518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.786488    2582 generic.go:356] "Generic (PLEG): container finished" podID="59ab844c-a9b2-435b-8bae-ee8e24813f12" containerID="59a5f536979e5970d21897f1dab69ee472a33c18a93b2218fe8c18e1484ddbf9" exitCode=0
Sep 11 21:48:53.786687 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.786586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" event={"ID":"59ab844c-a9b2-435b-8bae-ee8e24813f12","Type":"ContainerDied","Data":"59a5f536979e5970d21897f1dab69ee472a33c18a93b2218fe8c18e1484ddbf9"}
Sep 11 21:48:53.788232 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.788210    2582 generic.go:356] "Generic (PLEG): container finished" podID="b27dce4b-051c-4fdd-bce7-92989be07ee0" containerID="00a1c2085c063003a7e1b5094973bbe824995795a0b72d2e08dc13bcb6fb0c7a" exitCode=0
Sep 11 21:48:53.788338 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.788241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" event={"ID":"b27dce4b-051c-4fdd-bce7-92989be07ee0","Type":"ContainerDied","Data":"00a1c2085c063003a7e1b5094973bbe824995795a0b72d2e08dc13bcb6fb0c7a"}
Sep 11 21:48:53.810485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.810424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:53.869395 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.869327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:53.925209 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.925180    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:53.952551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.952517    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:53.975419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.975376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs\") pod \"b27dce4b-051c-4fdd-bce7-92989be07ee0\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") "
Sep 11 21:48:53.975622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.975449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmk79\" (UniqueName: \"kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79\") pod \"b27dce4b-051c-4fdd-bce7-92989be07ee0\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") "
Sep 11 21:48:53.975622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.975497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") pod \"b27dce4b-051c-4fdd-bce7-92989be07ee0\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") "
Sep 11 21:48:53.975622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.975561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca\") pod \"b27dce4b-051c-4fdd-bce7-92989be07ee0\" (UID: \"b27dce4b-051c-4fdd-bce7-92989be07ee0\") "
Sep 11 21:48:53.976229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.976183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b27dce4b-051c-4fdd-bce7-92989be07ee0" (UID: "b27dce4b-051c-4fdd-bce7-92989be07ee0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:53.978505 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.978471    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b27dce4b-051c-4fdd-bce7-92989be07ee0" (UID: "b27dce4b-051c-4fdd-bce7-92989be07ee0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:53.978624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.978573    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b27dce4b-051c-4fdd-bce7-92989be07ee0" (UID: "b27dce4b-051c-4fdd-bce7-92989be07ee0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:48:53.978975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:53.978946    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79" (OuterVolumeSpecName: "kube-api-access-mmk79") pod "b27dce4b-051c-4fdd-bce7-92989be07ee0" (UID: "b27dce4b-051c-4fdd-bce7-92989be07ee0"). InnerVolumeSpecName "kube-api-access-mmk79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:54.077202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c5tl\" (UniqueName: \"kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl\") pod \"59ab844c-a9b2-435b-8bae-ee8e24813f12\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") "
Sep 11 21:48:54.077202 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume\") pod \"59ab844c-a9b2-435b-8bae-ee8e24813f12\" (UID: \"59ab844c-a9b2-435b-8bae-ee8e24813f12\") "
Sep 11 21:48:54.077410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077354    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b27dce4b-051c-4fdd-bce7-92989be07ee0-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.077410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077365    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.077410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077374    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmk79\" (UniqueName: \"kubernetes.io/projected/b27dce4b-051c-4fdd-bce7-92989be07ee0-kube-api-access-mmk79\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.077410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077382    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b27dce4b-051c-4fdd-bce7-92989be07ee0-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.077622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.077600    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59ab844c-a9b2-435b-8bae-ee8e24813f12" (UID: "59ab844c-a9b2-435b-8bae-ee8e24813f12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:48:54.079391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.079371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl" (OuterVolumeSpecName: "kube-api-access-5c5tl") pod "59ab844c-a9b2-435b-8bae-ee8e24813f12" (UID: "59ab844c-a9b2-435b-8bae-ee8e24813f12"). InnerVolumeSpecName "kube-api-access-5c5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:48:54.168841 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.168741    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:54.177872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.177847    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5tl\" (UniqueName: \"kubernetes.io/projected/59ab844c-a9b2-435b-8bae-ee8e24813f12-kube-api-access-5c5tl\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.177872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.177872    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ab844c-a9b2-435b-8bae-ee8e24813f12-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:48:54.271745 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.271675    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:54.799606 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.799573    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"
Sep 11 21:48:54.800091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.799573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6" event={"ID":"59ab844c-a9b2-435b-8bae-ee8e24813f12","Type":"ContainerDied","Data":"bd1891925081caba6fd7a8389c2c25c5fd8741023f8621da62053491a407eb21"}
Sep 11 21:48:54.800091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.799761    2582 scope.go:117] "RemoveContainer" containerID="59a5f536979e5970d21897f1dab69ee472a33c18a93b2218fe8c18e1484ddbf9"
Sep 11 21:48:54.801618 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.801589    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"
Sep 11 21:48:54.802111 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.801606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj" event={"ID":"b27dce4b-051c-4fdd-bce7-92989be07ee0","Type":"ContainerDied","Data":"048b788b9cca0e62dec60bcc4cbafe4bbb6c9cc4d93fa79e162fc1a6d1a08138"}
Sep 11 21:48:54.809822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.809805    2582 scope.go:117] "RemoveContainer" containerID="00a1c2085c063003a7e1b5094973bbe824995795a0b72d2e08dc13bcb6fb0c7a"
Sep 11 21:48:54.817314 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.817291    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:54.821463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.821441    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a21586a5-755bdf4c97-4hgk6"]
Sep 11 21:48:54.831321 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.831300    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:54.836211 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:54.836189    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a21586a5-d56f85f87-bkrsj"]
Sep 11 21:48:55.207175 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:55.207081    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:55.909914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:55.909876    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:56.455406 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:56.455375    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ab844c-a9b2-435b-8bae-ee8e24813f12" path="/var/lib/kubelet/pods/59ab844c-a9b2-435b-8bae-ee8e24813f12/volumes"
Sep 11 21:48:56.455784 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:56.455771    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27dce4b-051c-4fdd-bce7-92989be07ee0" path="/var/lib/kubelet/pods/b27dce4b-051c-4fdd-bce7-92989be07ee0/volumes"
Sep 11 21:48:56.778181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:56.778152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:57.478914 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:57.478885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:48:59.944637 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:48:59.944601    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:00.045513 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:00.045470    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:00.205381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:00.205301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:00.553015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:00.552976    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:00.650785 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:00.650743    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:00.758293 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:00.758259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:06.640753 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:06.640707    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:06.741503 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:06.741463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:06.840888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:06.840848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:06.959393 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:06.959296    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:07.304855 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:07.304817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:11.209125 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:11.209077    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:12.942875 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:12.942841    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:13.042665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:13.042634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:13.825764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:13.825727    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:14.123040 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:14.122939    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:14.839292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:14.839258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:16.806451 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:16.806416    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:17.506149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:17.506109    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:17.624662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:17.624628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:17.722292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:17.722257    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:18.165428 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:18.165392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:49:18.545153 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:18.545110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:19.385697 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:19.385658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:19.778343 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:19.778313    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:20.037332 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.037241    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:20.197489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.197452    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:20.573440 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.573403    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:20.734372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.734336    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:49:20.735020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.734998    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27dce4b-051c-4fdd-bce7-92989be07ee0" containerName="maas-api"
Sep 11 21:49:20.735020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735020    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27dce4b-051c-4fdd-bce7-92989be07ee0" containerName="maas-api"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735035    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" containerName="payload-processing"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735044    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" containerName="payload-processing"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735061    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ab844c-a9b2-435b-8bae-ee8e24813f12" containerName="payload-processing"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735070    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ab844c-a9b2-435b-8bae-ee8e24813f12" containerName="payload-processing"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735087    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" containerName="maas-api"
Sep 11 21:49:20.735195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735095    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" containerName="maas-api"
Sep 11 21:49:20.735480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735214    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ab844c-a9b2-435b-8bae-ee8e24813f12" containerName="payload-processing"
Sep 11 21:49:20.735480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735226    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27dce4b-051c-4fdd-bce7-92989be07ee0" containerName="maas-api"
Sep 11 21:49:20.735480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735238    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2104f1a-0e01-43a5-a5c9-d342afd680dd" containerName="payload-processing"
Sep 11 21:49:20.735480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.735247    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8acb32d-3cfb-4c29-bc5f-f790df07ee9d" containerName="maas-api"
Sep 11 21:49:20.738800 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.738776    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.741145 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.741123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d3fe19-openshift-default-dockercfg-crdmb\""
Sep 11 21:49:20.741504 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.741469    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:49:20.776860 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.776824    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:20.825069 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.824975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825069 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825069 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgxj\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825179    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.825514 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.825351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.925824 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.925824 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgxj\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.925951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926503 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926503 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926614 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.926698 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.926677    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.928286 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.928259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.928674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.928651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.935282 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.935257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.935370 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.935310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgxj\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj\") pod \"e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") " pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:20.983947 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.983912    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:49:20.989204 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.989179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:20.991129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.991100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5cb5f4-serving-cert\""
Sep 11 21:49:20.997415 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:20.997242    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:49:21.016752 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.016706    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.026908 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.026885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kml\" (UniqueName: \"kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.027057 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.026918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.027057 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.026959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.027057 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.026989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.050486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.050465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:21.117655 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.117175    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.128067 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.128039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.128232 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.128170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kml\" (UniqueName: \"kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.128232 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.128211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.128343 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.128259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.128549 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:49:21.128509    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-serving-cert: secret "maas-api-e2e-shared-a-w2-5cb5f4-serving-cert" not found
Sep 11 21:49:21.128679 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:49:21.128576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls podName:dc405f57-f3e2-4707-aedf-f97d09b6c3f1 nodeName:}" failed. No retries permitted until 2026-09-11 21:49:21.628560939 +0000 UTC m=+1127.591585567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" (UID: "dc405f57-f3e2-4707-aedf-f97d09b6c3f1") : secret "maas-api-e2e-shared-a-w2-5cb5f4-serving-cert" not found
Sep 11 21:49:21.129224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.129144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.131106 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.131087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.138995 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.138633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kml\" (UniqueName: \"kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.217620 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.217579    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.271665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.271614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.331762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.331737    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:49:21.333631 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:21.333608    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b841c3a_48d5_49ec_a875_735a18e75788.slice/crio-73b279034a502774c2207cc525d24412753ddd89b65eb8dcfde02d6e7c947cbb WatchSource:0}: Error finding container 73b279034a502774c2207cc525d24412753ddd89b65eb8dcfde02d6e7c947cbb: Status 404 returned error can't find the container with id 73b279034a502774c2207cc525d24412753ddd89b65eb8dcfde02d6e7c947cbb
Sep 11 21:49:21.562409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.562372    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.634624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.634592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.637163 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.637145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.667100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.667068    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:49:21.671920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.671902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.672984 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.672959    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:21.673947 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.673928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5cb5f4\""
Sep 11 21:49:21.684099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.684077    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:49:21.735791 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.735757    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.735981 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.735804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn4x\" (UniqueName: \"kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.741613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.741581    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:49:21.745022 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.745006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.746979 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.746958    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5cb5f4-dockercfg-jpk74\""
Sep 11 21:49:21.755390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.755370    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:49:21.836774 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.836661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.836774 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.836746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn4x\" (UniqueName: \"kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.836993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.836811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.836993 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.836840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j84f8\" (UniqueName: \"kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.837370 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.837351    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.844409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.844390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn4x\" (UniqueName: \"kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x\") pod \"payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:21.902587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.902554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:21.938404 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.938359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.938582 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.938413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j84f8\" (UniqueName: \"kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.939140 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.939115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.946680 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.946653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j84f8\" (UniqueName: \"kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8\") pod \"payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:21.950261 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.950224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" event={"ID":"2b841c3a-48d5-49ec-a875-735a18e75788","Type":"ContainerStarted","Data":"73b279034a502774c2207cc525d24412753ddd89b65eb8dcfde02d6e7c947cbb"}
Sep 11 21:49:21.983854 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:21.983820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:22.055435 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.055332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:22.180237 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.180211    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:49:22.182019 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:22.181991    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc405f57_f3e2_4707_aedf_f97d09b6c3f1.slice/crio-eeb2c92d4408126a5db1c28470e66e0794cfec679545ab242b0a1f4886438803 WatchSource:0}: Error finding container eeb2c92d4408126a5db1c28470e66e0794cfec679545ab242b0a1f4886438803: Status 404 returned error can't find the container with id eeb2c92d4408126a5db1c28470e66e0794cfec679545ab242b0a1f4886438803
Sep 11 21:49:22.263797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.263770    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:49:22.265867 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:22.265835    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a987da3_932a_4dc6_931a_72089eae4097.slice/crio-6ab4e4c8136ad27860e2ca14625fa38d375c4e3f0a5c0a60c7fdd3dad5739f9c WatchSource:0}: Error finding container 6ab4e4c8136ad27860e2ca14625fa38d375c4e3f0a5c0a60c7fdd3dad5739f9c: Status 404 returned error can't find the container with id 6ab4e4c8136ad27860e2ca14625fa38d375c4e3f0a5c0a60c7fdd3dad5739f9c
Sep 11 21:49:22.350833 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.350757    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:49:22.355343 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:22.355305    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd834416f_4984_4e12_bb78_7aad45363ad1.slice/crio-5e39eedc3b860b81cdce14bf4bc0090e48c165cdf62e6e58e8154be2d482c52a WatchSource:0}: Error finding container 5e39eedc3b860b81cdce14bf4bc0090e48c165cdf62e6e58e8154be2d482c52a: Status 404 returned error can't find the container with id 5e39eedc3b860b81cdce14bf4bc0090e48c165cdf62e6e58e8154be2d482c52a
Sep 11 21:49:22.960179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.960091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" event={"ID":"d834416f-4984-4e12-bb78-7aad45363ad1","Type":"ContainerStarted","Data":"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"}
Sep 11 21:49:22.960179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.960156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" event={"ID":"d834416f-4984-4e12-bb78-7aad45363ad1","Type":"ContainerStarted","Data":"5e39eedc3b860b81cdce14bf4bc0090e48c165cdf62e6e58e8154be2d482c52a"}
Sep 11 21:49:22.960667 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.960305    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:22.962353 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.962329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" event={"ID":"1a987da3-932a-4dc6-931a-72089eae4097","Type":"ContainerStarted","Data":"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"}
Sep 11 21:49:22.962505 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.962372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" event={"ID":"1a987da3-932a-4dc6-931a-72089eae4097","Type":"ContainerStarted","Data":"6ab4e4c8136ad27860e2ca14625fa38d375c4e3f0a5c0a60c7fdd3dad5739f9c"}
Sep 11 21:49:22.962774 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.962755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:22.964597 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.964562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" event={"ID":"dc405f57-f3e2-4707-aedf-f97d09b6c3f1","Type":"ContainerStarted","Data":"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"}
Sep 11 21:49:22.964694 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.964617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:22.964694 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.964643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" event={"ID":"dc405f57-f3e2-4707-aedf-f97d09b6c3f1","Type":"ContainerStarted","Data":"eeb2c92d4408126a5db1c28470e66e0794cfec679545ab242b0a1f4886438803"}
Sep 11 21:49:22.976983 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.976914    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" podStartSLOduration=1.976898529 podStartE2EDuration="1.976898529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:22.975913087 +0000 UTC m=+1128.938937737" watchObservedRunningTime="2026-09-11 21:49:22.976898529 +0000 UTC m=+1128.939923177"
Sep 11 21:49:22.992316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:22.992250    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" podStartSLOduration=1.992231361 podStartE2EDuration="1.992231361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:22.990477996 +0000 UTC m=+1128.953502657" watchObservedRunningTime="2026-09-11 21:49:22.992231361 +0000 UTC m=+1128.955256012"
Sep 11 21:49:23.010112 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:23.010050    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" podStartSLOduration=2.587267737 podStartE2EDuration="3.010028113s" totalImagesPullingTime="422.760376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:23.00629029 +0000 UTC m=+1128.969314939" watchObservedRunningTime="2026-09-11 21:49:23.010028113 +0000 UTC m=+1128.973052764"
Sep 11 21:49:24.099267 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:24.099228    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:49:24.099552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:24.099316    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:49:24.099552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:24.099347    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:49:24.978222 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:24.978185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" event={"ID":"2b841c3a-48d5-49ec-a875-735a18e75788","Type":"ContainerStarted","Data":"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"}
Sep 11 21:49:25.010221 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.010169    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" podStartSLOduration=2.247030354 podStartE2EDuration="5.010152445s" totalImagesPullingTime="2.763122091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:25.008429379 +0000 UTC m=+1130.971454030" watchObservedRunningTime="2026-09-11 21:49:25.010152445 +0000 UTC m=+1130.973177093"
Sep 11 21:49:25.051548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.051510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:25.057191 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.057163    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:25.344777 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.344660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:25.443469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.443432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:25.545239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.545192    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:25.644316 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.644222    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:25.984097 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.984064    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:25.985260 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:25.985241    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:49:28.976776 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:28.976745    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:49:33.974137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:33.974108    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:49:34.980930 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:34.980899    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:49:37.689781 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:37.689742    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:42.090158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:42.090111    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:42.205623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:42.205585    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:43.323442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:43.323398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:43.990338 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:43.990305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:44.802215 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:44.802177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:45.493561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:45.493525    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:46.216494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:46.216454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:47.130256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:47.130218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:47.685555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:47.685516    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:47.780081 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:47.780040    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:49:48.007485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:48.007445    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:48.312085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:48.311998    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:48.746538 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:48.746500    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:51.178062 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:51.178026    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:51.797186 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:51.797139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:51.897657 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:51.897610    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:51.997751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:51.997693    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:52.732203 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:52.732174    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:53.383006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:53.382964    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:54.099561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:54.099528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:54.710777 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:54.710744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:55.513864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:55.513829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:55.588771 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:55.588697    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:55.979051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:55.979019    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:56.076109 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.076070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:56.082029 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.082000    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:49:56.091159 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.091131    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.093387 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.093354    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-afd794\""
Sep 11 21:49:56.093528 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.093405    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-afd794-dockercfg-27z89\""
Sep 11 21:49:56.094554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.094531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:49:56.115831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.115801    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:49:56.120283 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.120259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.130057 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.130025    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:49:56.147901 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.147868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:49:56.271425 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.271330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftqb\" (UniqueName: \"kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.271425 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.271404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msljl\" (UniqueName: \"kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.271624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.271429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.271624 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.271535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.372586 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.372530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msljl\" (UniqueName: \"kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.372816 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.372594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.372816 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.372687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.372816 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.372756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftqb\" (UniqueName: \"kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.373357 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.373334    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.373426 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.373343    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.381117 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.381091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftqb\" (UniqueName: \"kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb\") pod \"payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.381221 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.381195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msljl\" (UniqueName: \"kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl\") pod \"payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.403190 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.403159    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:56.432181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.432152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:56.677016 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.676981    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:49:56.679100 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:56.679077    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965cdd75_56af_409e_a4dc_8e6060dae9a6.slice/crio-e10c2ccd0f2c4a01a857066dfdd7e187289a29fb2962ec70fba9142f7fb5dcee WatchSource:0}: Error finding container e10c2ccd0f2c4a01a857066dfdd7e187289a29fb2962ec70fba9142f7fb5dcee: Status 404 returned error can't find the container with id e10c2ccd0f2c4a01a857066dfdd7e187289a29fb2962ec70fba9142f7fb5dcee
Sep 11 21:49:56.715729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:56.715687    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:49:56.717895 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:49:56.717869    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6946fab_2dd3_4f86_877d_247165a21c19.slice/crio-119a749a39bbe0823d7ee948558f64c941d44edf721ed40f53d8662a18cc4046 WatchSource:0}: Error finding container 119a749a39bbe0823d7ee948558f64c941d44edf721ed40f53d8662a18cc4046: Status 404 returned error can't find the container with id 119a749a39bbe0823d7ee948558f64c941d44edf721ed40f53d8662a18cc4046
Sep 11 21:49:57.147636 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.147597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" event={"ID":"b6946fab-2dd3-4f86-877d-247165a21c19","Type":"ContainerStarted","Data":"c23d62166a01594d884467662f84f92ff908e105f1c274842617974fca8d9482"}
Sep 11 21:49:57.147879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.147748    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:49:57.147879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.147789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" event={"ID":"b6946fab-2dd3-4f86-877d-247165a21c19","Type":"ContainerStarted","Data":"119a749a39bbe0823d7ee948558f64c941d44edf721ed40f53d8662a18cc4046"}
Sep 11 21:49:57.149362 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.149336    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" event={"ID":"965cdd75-56af-409e-a4dc-8e6060dae9a6","Type":"ContainerStarted","Data":"82eae41930a79bfeb261b1823d67695fae187c9843ccb1f77c639c1d8edd19be"}
Sep 11 21:49:57.149489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.149389    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:49:57.149489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.149415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" event={"ID":"965cdd75-56af-409e-a4dc-8e6060dae9a6","Type":"ContainerStarted","Data":"e10c2ccd0f2c4a01a857066dfdd7e187289a29fb2962ec70fba9142f7fb5dcee"}
Sep 11 21:49:57.163692 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.163633    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" podStartSLOduration=1.163616111 podStartE2EDuration="1.163616111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:57.161972469 +0000 UTC m=+1163.124997123" watchObservedRunningTime="2026-09-11 21:49:57.163616111 +0000 UTC m=+1163.126640757"
Sep 11 21:49:57.178148 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:49:57.178098    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" podStartSLOduration=1.178081534 podStartE2EDuration="1.178081534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:49:57.176181411 +0000 UTC m=+1163.139206060" watchObservedRunningTime="2026-09-11 21:49:57.178081534 +0000 UTC m=+1163.141106180"
Sep 11 21:50:07.778287 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:07.778250    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:08.160141 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:08.160063    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:50:08.160141 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:08.160131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:50:10.962409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:10.962368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:50:17.492056 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:17.491984    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:18.128013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:18.127979    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:18.782505 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:18.782456    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:19.579348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:19.579315    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:20.215548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:20.215508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:20.979031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:20.978994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:21.682138 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:21.682105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:21.903813 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:21.903779    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:23.755455 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:23.755410    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:31.455766 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.455726    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:31.720864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.720788    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:50:31.724574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.724545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.726749 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.726705    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9f68da5e-openshift-default-dockercfg-n56xf\""
Sep 11 21:50:31.735642 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.735615    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:50:31.745888 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.745858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:31.824272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824447 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824447 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824447 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.824696 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.824677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458vm\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.854479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.854440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:31.925664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.925889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.925997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458vm\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.926212    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926370 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.926316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926426 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.926380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926488 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.926462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.926704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.926617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.928239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.928215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.928497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.928478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.937583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.937548    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.938334 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.938304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-458vm\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm\") pod \"e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") " pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:31.942844 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:31.942812    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:32.037257 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.037217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:32.316308 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.316281    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:50:32.318540 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:50:32.318517    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c0d014_7b38_42e3_b5d7_4a34264687c0.slice/crio-476ddb0e91f455e1f563fc2cf4742dd9eca6e6a0bf826eb2e1d8dc1c70005c8f WatchSource:0}: Error finding container 476ddb0e91f455e1f563fc2cf4742dd9eca6e6a0bf826eb2e1d8dc1c70005c8f: Status 404 returned error can't find the container with id 476ddb0e91f455e1f563fc2cf4742dd9eca6e6a0bf826eb2e1d8dc1c70005c8f
Sep 11 21:50:32.320908 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.320877    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:50:32.320986 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.320948    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:50:32.320986 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.320977    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:50:32.353190 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.353159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerStarted","Data":"476ddb0e91f455e1f563fc2cf4742dd9eca6e6a0bf826eb2e1d8dc1c70005c8f"}
Sep 11 21:50:32.711885 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:32.711788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:33.360729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:33.360691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerStarted","Data":"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"}
Sep 11 21:50:33.386099 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:33.386041    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podStartSLOduration=2.386024339 podStartE2EDuration="2.386024339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:50:33.383834952 +0000 UTC m=+1199.346859624" watchObservedRunningTime="2026-09-11 21:50:33.386024339 +0000 UTC m=+1199.349048988"
Sep 11 21:50:33.388332 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:33.388297    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:34.037516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:34.037475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:34.043178 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:34.043152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:34.224520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:34.224487    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:34.367043 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:34.366951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:34.367963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:34.367944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:50:35.176145 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:35.176105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:35.403623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:35.403588    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:35.503321 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:35.503278    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:35.912470 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:35.912383    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:36.698762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:36.698710    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:36.767625 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:36.767575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:50:37.376689 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:37.376647    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.019467 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.019393    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.119156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.119121    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.220271 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.220234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.468427 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.468314    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.590696 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.590664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:38.919905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:38.919862    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:39.609612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:39.609580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:40.429415 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:40.429363    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.076755 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.076699    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.591941 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.591894    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:50:41.592218 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.592187    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" podUID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" containerName="maas-api" containerID="cri-o://5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67" gracePeriod=30
Sep 11 21:50:41.614497 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.614469    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"]
Sep 11 21:50:41.619729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.619678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.620070 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.620049    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e919c5-29819385-w6hpk"]
Sep 11 21:50:41.643035 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.643000    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:50:41.643306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.643255    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" podUID="8ea32560-d251-4109-9b55-a9a6387e4ed2" containerName="payload-processing" containerID="cri-o://0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430" gracePeriod=30
Sep 11 21:50:41.650498 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.650470    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:50:41.651236 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.651200    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" podUID="c9380e66-d701-4874-b0e9-a139941233ea" containerName="payload-pre-processing" containerID="cri-o://ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7" gracePeriod=30
Sep 11 21:50:41.685075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.685033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.785444 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.785392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.890308 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.890274    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:41.943102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.943072    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:50:41.956245 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.956214    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:50:41.967647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:41.967624    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:50:42.025385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025354    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrs8p\" (UniqueName: \"kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p\") pod \"c9380e66-d701-4874-b0e9-a139941233ea\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") "
Sep 11 21:50:42.025592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025402    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") pod \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") "
Sep 11 21:50:42.025592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025460    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume\") pod \"c9380e66-d701-4874-b0e9-a139941233ea\" (UID: \"c9380e66-d701-4874-b0e9-a139941233ea\") "
Sep 11 21:50:42.025592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025506    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pt87\" (UniqueName: \"kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87\") pod \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") "
Sep 11 21:50:42.025592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k7tz\" (UniqueName: \"kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz\") pod \"8ea32560-d251-4109-9b55-a9a6387e4ed2\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") "
Sep 11 21:50:42.025913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca\") pod \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") "
Sep 11 21:50:42.025913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025658    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume\") pod \"8ea32560-d251-4109-9b55-a9a6387e4ed2\" (UID: \"8ea32560-d251-4109-9b55-a9a6387e4ed2\") "
Sep 11 21:50:42.025913 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs\") pod \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\" (UID: \"c9a17eb6-8480-439d-a313-9fcd0bd24c33\") "
Sep 11 21:50:42.027709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.025933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9380e66-d701-4874-b0e9-a139941233ea" (UID: "c9380e66-d701-4874-b0e9-a139941233ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:42.027709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.026769    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9a17eb6-8480-439d-a313-9fcd0bd24c33" (UID: "c9a17eb6-8480-439d-a313-9fcd0bd24c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:42.027709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.026710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ea32560-d251-4109-9b55-a9a6387e4ed2" (UID: "8ea32560-d251-4109-9b55-a9a6387e4ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:50:42.029454 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.029425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz" (OuterVolumeSpecName: "kube-api-access-8k7tz") pod "8ea32560-d251-4109-9b55-a9a6387e4ed2" (UID: "8ea32560-d251-4109-9b55-a9a6387e4ed2"). InnerVolumeSpecName "kube-api-access-8k7tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:50:42.029592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.029458    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9a17eb6-8480-439d-a313-9fcd0bd24c33" (UID: "c9a17eb6-8480-439d-a313-9fcd0bd24c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:42.029592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.029464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p" (OuterVolumeSpecName: "kube-api-access-lrs8p") pod "c9380e66-d701-4874-b0e9-a139941233ea" (UID: "c9380e66-d701-4874-b0e9-a139941233ea"). InnerVolumeSpecName "kube-api-access-lrs8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:50:42.029744 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.029681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87" (OuterVolumeSpecName: "kube-api-access-9pt87") pod "c9a17eb6-8480-439d-a313-9fcd0bd24c33" (UID: "c9a17eb6-8480-439d-a313-9fcd0bd24c33"). InnerVolumeSpecName "kube-api-access-9pt87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:50:42.030486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.030464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9a17eb6-8480-439d-a313-9fcd0bd24c33" (UID: "c9a17eb6-8480-439d-a313-9fcd0bd24c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127172    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9380e66-d701-4874-b0e9-a139941233ea-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127205    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pt87\" (UniqueName: \"kubernetes.io/projected/c9a17eb6-8480-439d-a313-9fcd0bd24c33-kube-api-access-9pt87\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127218    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7tz\" (UniqueName: \"kubernetes.io/projected/8ea32560-d251-4109-9b55-a9a6387e4ed2-kube-api-access-8k7tz\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127229    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9a17eb6-8480-439d-a313-9fcd0bd24c33-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127241    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ea32560-d251-4109-9b55-a9a6387e4ed2-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127255    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127268    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrs8p\" (UniqueName: \"kubernetes.io/projected/c9380e66-d701-4874-b0e9-a139941233ea-kube-api-access-lrs8p\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.127867 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.127280    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9a17eb6-8480-439d-a313-9fcd0bd24c33-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:50:42.163687 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.163650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:42.264814 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.264783    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:42.423025 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.422925    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" containerID="5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67" exitCode=0
Sep 11 21:50:42.423025 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.422979    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"
Sep 11 21:50:42.423025 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.423001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" event={"ID":"c9a17eb6-8480-439d-a313-9fcd0bd24c33","Type":"ContainerDied","Data":"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"}
Sep 11 21:50:42.423336 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.423092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk" event={"ID":"c9a17eb6-8480-439d-a313-9fcd0bd24c33","Type":"ContainerDied","Data":"eb90ec4fc62f77a98429ba96cabd18974c1d6a04e6c99c20831ab4a97b1a40b8"}
Sep 11 21:50:42.423336 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.423122    2582 scope.go:117] "RemoveContainer" containerID="5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"
Sep 11 21:50:42.424552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.424528    2582 generic.go:356] "Generic (PLEG): container finished" podID="8ea32560-d251-4109-9b55-a9a6387e4ed2" containerID="0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430" exitCode=0
Sep 11 21:50:42.424673 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.424596    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"
Sep 11 21:50:42.424673 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.424606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" event={"ID":"8ea32560-d251-4109-9b55-a9a6387e4ed2","Type":"ContainerDied","Data":"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"}
Sep 11 21:50:42.424673 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.424651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv" event={"ID":"8ea32560-d251-4109-9b55-a9a6387e4ed2","Type":"ContainerDied","Data":"2050d777fb3c98b91be2e6424e3f674e0f3bbce9c049d96ffd85b3aaefb167c4"}
Sep 11 21:50:42.426256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.426233    2582 generic.go:356] "Generic (PLEG): container finished" podID="c9380e66-d701-4874-b0e9-a139941233ea" containerID="ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7" exitCode=0
Sep 11 21:50:42.426378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.426274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" event={"ID":"c9380e66-d701-4874-b0e9-a139941233ea","Type":"ContainerDied","Data":"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"}
Sep 11 21:50:42.426378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.426292    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"
Sep 11 21:50:42.426378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.426314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg" event={"ID":"c9380e66-d701-4874-b0e9-a139941233ea","Type":"ContainerDied","Data":"b1477e05e21b217a23ca11f373e5ff112dc2b15d19c625a1abd1596e7c3a7039"}
Sep 11 21:50:42.433566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.433549    2582 scope.go:117] "RemoveContainer" containerID="5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"
Sep 11 21:50:42.433957 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:50:42.433937    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67\": container with ID starting with 5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67 not found: ID does not exist" containerID="5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"
Sep 11 21:50:42.434047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.433970    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67"} err="failed to get container status \"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67\": rpc error: code = NotFound desc = could not find container \"5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67\": container with ID starting with 5a253a30a036344a9737457870e02024c86528891856c1bdf8c7a252e941cb67 not found: ID does not exist"
Sep 11 21:50:42.434047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.433993    2582 scope.go:117] "RemoveContainer" containerID="0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"
Sep 11 21:50:42.442440 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.442425    2582 scope.go:117] "RemoveContainer" containerID="0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"
Sep 11 21:50:42.442744 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:50:42.442707    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430\": container with ID starting with 0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430 not found: ID does not exist" containerID="0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"
Sep 11 21:50:42.442783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.442752    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430"} err="failed to get container status \"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430\": rpc error: code = NotFound desc = could not find container \"0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430\": container with ID starting with 0d0cad6222dc3daf42f44a60aed8e155ff21f1cb1b19f4fb6638191ff8f65430 not found: ID does not exist"
Sep 11 21:50:42.442783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.442768    2582 scope.go:117] "RemoveContainer" containerID="ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"
Sep 11 21:50:42.447516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.447495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:50:42.452920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.452897    2582 scope.go:117] "RemoveContainer" containerID="ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"
Sep 11 21:50:42.453249 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:50:42.453229    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7\": container with ID starting with ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7 not found: ID does not exist" containerID="ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"
Sep 11 21:50:42.453297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.453256    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7"} err="failed to get container status \"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7\": rpc error: code = NotFound desc = could not find container \"ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7\": container with ID starting with ee6ac81587abe235f8ef3b4537ffd8ace0dd3f1aeea35e53be33ef0e3a2130a7 not found: ID does not exist"
Sep 11 21:50:42.455331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.455312    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa07be3-8db2-4346-9b8c-01bd49506205" path="/var/lib/kubelet/pods/7fa07be3-8db2-4346-9b8c-01bd49506205/volumes"
Sep 11 21:50:42.455546 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.455534    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e919c5-79cd64c6c6-g6hhk"]
Sep 11 21:50:42.462012 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.461992    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:50:42.465508 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.465488    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e919c5-6d8485488f-mkpfv"]
Sep 11 21:50:42.476047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.476029    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:50:42.481897 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:42.481878    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e919c5-65978b484b-pnzdg"]
Sep 11 21:50:43.101261 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:43.101226    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:43.767574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:43.767538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:44.454416 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:44.454388    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea32560-d251-4109-9b55-a9a6387e4ed2" path="/var/lib/kubelet/pods/8ea32560-d251-4109-9b55-a9a6387e4ed2/volumes"
Sep 11 21:50:44.454755 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:44.454740    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9380e66-d701-4874-b0e9-a139941233ea" path="/var/lib/kubelet/pods/c9380e66-d701-4874-b0e9-a139941233ea/volumes"
Sep 11 21:50:44.455083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:44.455072    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" path="/var/lib/kubelet/pods/c9a17eb6-8480-439d-a313-9fcd0bd24c33/volumes"
Sep 11 21:50:44.573822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:44.573787    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:45.235561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:45.235496    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:45.895384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:45.895347    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:46.694997 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:46.694961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:47.194878 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:47.194833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:47.369500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:47.369468    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:58.155608 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.155574    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:50:58.156114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.155808    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" podUID="2b841c3a-48d5-49ec-a875-735a18e75788" containerName="istio-proxy" containerID="cri-o://98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55" gracePeriod=30
Sep 11 21:50:58.179687 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.179654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:58.406850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.406751    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:58.542385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.542333    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:50:58.935515 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:50:58.935474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:00.263264 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:00.263228    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:00.364048 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:00.364011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:01.092025 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:01.091987    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:01.785473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:01.785433    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:02.595130 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:02.595090    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:03.317168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.317131    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:03.417537 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.417472    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:51:03.551412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.551374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:03.555611 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.555808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.555808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.555808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555691    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.555808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.555808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555799    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.556085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555924    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.556085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:03.556085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.555948    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:03.556085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvgxj\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.556085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy\") pod \"2b841c3a-48d5-49ec-a875-735a18e75788\" (UID: \"2b841c3a-48d5-49ec-a875-735a18e75788\") "
Sep 11 21:51:03.556319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556077    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:03.556319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556151    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:03.556319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556232    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:51:03.556555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556533    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b841c3a-48d5-49ec-a875-735a18e75788-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.556669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556559    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.556669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556581    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.556669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556592    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.556669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.556604    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.558674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.558633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:03.558674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.558655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:51:03.558674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.558659    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj" (OuterVolumeSpecName: "kube-api-access-rvgxj") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "kube-api-access-rvgxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:03.559600 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.559580    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b841c3a-48d5-49ec-a875-735a18e75788" (UID: "2b841c3a-48d5-49ec-a875-735a18e75788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:51:03.560143 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.560118    2582 generic.go:356] "Generic (PLEG): container finished" podID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerID="90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a" exitCode=0
Sep 11 21:51:03.560240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.560184    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerDied","Data":"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"}
Sep 11 21:51:03.560695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.560674    2582 scope.go:117] "RemoveContainer" containerID="90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"
Sep 11 21:51:03.561784 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.561760    2582 generic.go:356] "Generic (PLEG): container finished" podID="2b841c3a-48d5-49ec-a875-735a18e75788" containerID="98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55" exitCode=0
Sep 11 21:51:03.561877 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.561815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" event={"ID":"2b841c3a-48d5-49ec-a875-735a18e75788","Type":"ContainerDied","Data":"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"}
Sep 11 21:51:03.561877 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.561819    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"
Sep 11 21:51:03.561877 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.561857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7" event={"ID":"2b841c3a-48d5-49ec-a875-735a18e75788","Type":"ContainerDied","Data":"73b279034a502774c2207cc525d24412753ddd89b65eb8dcfde02d6e7c947cbb"}
Sep 11 21:51:03.562047 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.561878    2582 scope.go:117] "RemoveContainer" containerID="98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"
Sep 11 21:51:03.562220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.562185    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:03.562330 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.562278    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:03.562394 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.562367    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:03.579066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.579043    2582 scope.go:117] "RemoveContainer" containerID="98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"
Sep 11 21:51:03.579410 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:51:03.579393    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55\": container with ID starting with 98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55 not found: ID does not exist" containerID="98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"
Sep 11 21:51:03.579475 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.579419    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55"} err="failed to get container status \"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55\": rpc error: code = NotFound desc = could not find container \"98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55\": container with ID starting with 98af9abcf5e9aece8b84e5a7a05f9ffa4d9f127524d8ea955bad6f93d9a1ad55 not found: ID does not exist"
Sep 11 21:51:03.597692 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.597659    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:51:03.601222 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.601196    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d3fe19-openshift-default-694bcc8f68-x7td7"]
Sep 11 21:51:03.657704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.657667    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvgxj\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-kube-api-access-rvgxj\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.657704 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.657699    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b841c3a-48d5-49ec-a875-735a18e75788-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.657932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.657744    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b841c3a-48d5-49ec-a875-735a18e75788-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.657932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.657757    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b841c3a-48d5-49ec-a875-735a18e75788-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:03.989879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:03.989842    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:04.367882 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:04.367794    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:51:04.455768 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:04.455734    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b841c3a-48d5-49ec-a875-735a18e75788" path="/var/lib/kubelet/pods/2b841c3a-48d5-49ec-a875-735a18e75788/volumes"
Sep 11 21:51:04.569235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:04.569201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerStarted","Data":"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"}
Sep 11 21:51:04.804960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:04.804926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:05.038239 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:05.038203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:51:05.039356 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:05.039329    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 21:51:05.039468 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:05.039395    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 21:51:06.038441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:06.038404    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 21:51:06.038872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:06.038500    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 21:51:07.038433 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:07.038394    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 21:51:07.038626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:07.038449    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 21:51:08.038664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:08.038618    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 21:51:08.039167 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:08.038682    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 21:51:09.037789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:09.037708    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 11 21:51:09.037988 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:09.037816    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 11 21:51:09.929317 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:09.929281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:10.043734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:10.043683    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:51:10.044194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:10.044178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:51:10.044906 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:10.044890    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:51:12.168645 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:12.168605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:51:23.551474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:23.551443    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:24.736173 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:24.736137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:34.557685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:34.557612    2582 scope.go:117] "RemoveContainer" containerID="bec4c68c57436db115611b95c7af0a1717bd22c36198f03c708c05fb6c89fb0c"
Sep 11 21:51:34.568492 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:34.568471    2582 scope.go:117] "RemoveContainer" containerID="843c285ecee3306d914f63e36eff89ac1f35c875c86cef902090a412c94bc769"
Sep 11 21:51:36.187612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:36.187572    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:38.805662 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:38.805627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:38.883201 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:38.883167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:39.199114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:39.199017    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:39.648575 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:39.648532    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:39.734418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:39.734379    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:40.613859 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:40.613821    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:41.503618 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:41.503580    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:41.646556 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:41.646522    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:42.247063 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.247016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:42.963352 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963317    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:51:42.963862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963846    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b841c3a-48d5-49ec-a875-735a18e75788" containerName="istio-proxy"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963864    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b841c3a-48d5-49ec-a875-735a18e75788" containerName="istio-proxy"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963874    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" containerName="maas-api"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963880    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" containerName="maas-api"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963889    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea32560-d251-4109-9b55-a9a6387e4ed2" containerName="payload-processing"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963896    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea32560-d251-4109-9b55-a9a6387e4ed2" containerName="payload-processing"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963922    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9380e66-d701-4874-b0e9-a139941233ea" containerName="payload-pre-processing"
Sep 11 21:51:42.963939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.963930    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9380e66-d701-4874-b0e9-a139941233ea" containerName="payload-pre-processing"
Sep 11 21:51:42.964181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.964007    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea32560-d251-4109-9b55-a9a6387e4ed2" containerName="payload-processing"
Sep 11 21:51:42.964181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.964017    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9380e66-d701-4874-b0e9-a139941233ea" containerName="payload-pre-processing"
Sep 11 21:51:42.964181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.964024    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9a17eb6-8480-439d-a313-9fcd0bd24c33" containerName="maas-api"
Sep 11 21:51:42.964181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.964030    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b841c3a-48d5-49ec-a875-735a18e75788" containerName="istio-proxy"
Sep 11 21:51:42.968822 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.968801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:42.971852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.971825    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f68da5e\"/\"default-dockercfg-l4djg\""
Sep 11 21:51:42.971985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.971831    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f68da5e\"/\"kube-root-ca.crt\""
Sep 11 21:51:42.971985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.971901    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f68da5e\"/\"openshift-service-ca.crt\""
Sep 11 21:51:42.971985 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.971835    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f68da5e\"/\"e2e-lifecycle-model-9f68da5e-kserve-self-signed-certs\""
Sep 11 21:51:42.976192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.976167    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:51:42.990364 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:42.990324    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:43.036831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.036791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.037013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.036858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.037013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.036953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.037013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.036977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.037013 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.037002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjlx\" (UniqueName: \"kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.037168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.037020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.091404 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.091368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:43.137729 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.137690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.137922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.137766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjlx\" (UniqueName: \"kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.137922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.137790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.137922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.137864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.138055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.137920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.138055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.138007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.138142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.138087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.138325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.138304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.138367 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.138333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.140263 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.140239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.140376 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.140357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.145457 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.145431    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjlx\" (UniqueName: \"kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx\") pod \"e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") " pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.279468 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.279423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:43.555270 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.555207    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:51:43.557684 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:51:43.557646    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf881d9_797c_47f0_8894_5e265e024477.slice/crio-b7711cc3b687ab3218785fe2eb6a4499b3ab889d6cf2b24ba79c7e037fcd9295 WatchSource:0}: Error finding container b7711cc3b687ab3218785fe2eb6a4499b3ab889d6cf2b24ba79c7e037fcd9295: Status 404 returned error can't find the container with id b7711cc3b687ab3218785fe2eb6a4499b3ab889d6cf2b24ba79c7e037fcd9295
Sep 11 21:51:43.780458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.780417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" event={"ID":"edf881d9-797c-47f0-8894-5e265e024477","Type":"ContainerStarted","Data":"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"}
Sep 11 21:51:43.780458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.780463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" event={"ID":"edf881d9-797c-47f0-8894-5e265e024477","Type":"ContainerStarted","Data":"b7711cc3b687ab3218785fe2eb6a4499b3ab889d6cf2b24ba79c7e037fcd9295"}
Sep 11 21:51:43.794126 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.794075    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:43.798852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:43.798802    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" podStartSLOduration=1.7987833 podStartE2EDuration="1.7987833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:51:43.796697647 +0000 UTC m=+1269.759722298" watchObservedRunningTime="2026-09-11 21:51:43.7987833 +0000 UTC m=+1269.761807949"
Sep 11 21:51:44.579057 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:44.579020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:45.171262 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:45.171224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:45.385035 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:45.384995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:47.190364 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:47.190331    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:47.266539 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:47.266475    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:47.372084 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:47.372048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:51:49.930352 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:49.930316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:49.958492 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:49.958454    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:51:49.958779 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:49.958749    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" podUID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" containerName="payload-processing" containerID="cri-o://3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb" gracePeriod=30
Sep 11 21:51:50.030538 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.030502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:50.134682 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.134589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:50.193474 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.193374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:50.241092 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.241064    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:51:50.409832 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.409800    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume\") pod \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") "
Sep 11 21:51:50.410019 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.409849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94jd9\" (UniqueName: \"kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9\") pod \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\" (UID: \"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547\") "
Sep 11 21:51:50.410221 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.410199    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" (UID: "6de58ae2-5cc9-4fbe-8350-5e7fb8b41547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:51:50.412059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.412040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9" (OuterVolumeSpecName: "kube-api-access-94jd9") pod "6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" (UID: "6de58ae2-5cc9-4fbe-8350-5e7fb8b41547"). InnerVolumeSpecName "kube-api-access-94jd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:51:50.483339 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.483304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:50.511518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.511484    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:50.511518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.511510    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94jd9\" (UniqueName: \"kubernetes.io/projected/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547-kube-api-access-94jd9\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:51:50.583041 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.583001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:50.816789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.816754    2582 generic.go:356] "Generic (PLEG): container finished" podID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" containerID="3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb" exitCode=0
Sep 11 21:51:50.816969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.816805    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"
Sep 11 21:51:50.816969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.816832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" event={"ID":"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547","Type":"ContainerDied","Data":"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"}
Sep 11 21:51:50.816969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.816884    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n" event={"ID":"6de58ae2-5cc9-4fbe-8350-5e7fb8b41547","Type":"ContainerDied","Data":"d60b03c1a8ca6f10ca1d15c6e37bb625aba77bec56eaf410be320bc230279ac5"}
Sep 11 21:51:50.816969 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.816909    2582 scope.go:117] "RemoveContainer" containerID="3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"
Sep 11 21:51:50.826023 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.826001    2582 scope.go:117] "RemoveContainer" containerID="3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"
Sep 11 21:51:50.826291 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:51:50.826272    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb\": container with ID starting with 3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb not found: ID does not exist" containerID="3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"
Sep 11 21:51:50.826344 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.826299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb"} err="failed to get container status \"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb\": rpc error: code = NotFound desc = could not find container \"3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb\": container with ID starting with 3b1c71f3c0c19ed404cb5469029f868c08822109e744823555fada3d5709cfbb not found: ID does not exist"
Sep 11 21:51:50.833623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.833599    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:51:50.838349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:50.838321    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7e6f5e-dcfcb84cf-gvh8n"]
Sep 11 21:51:51.387494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:51.387458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:52.078787 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:52.078709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:52.455658 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:52.455578    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" path="/var/lib/kubelet/pods/6de58ae2-5cc9-4fbe-8350-5e7fb8b41547/volumes"
Sep 11 21:51:52.767358 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:52.767326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:53.280031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:53.279999    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:53.280031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:53.280041    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:53.292829 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:53.292801    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:53.498974 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:53.498944    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:53.847427 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:53.847387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:51:54.133135 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:54.133043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:56.631416 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:56.631370    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:56.728494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:56.728457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:56.895249 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:56.895138    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:57.453424 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.453384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:57.722033 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.721951    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 21:51:57.722617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.722595    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" containerName="payload-processing"
Sep 11 21:51:57.722813 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.722792    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" containerName="payload-processing"
Sep 11 21:51:57.722966 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.722940    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de58ae2-5cc9-4fbe-8350-5e7fb8b41547" containerName="payload-processing"
Sep 11 21:51:57.727679 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.727660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.730429 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.730411    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-7d7821-openshift-default-dockercfg-gj86j\""
Sep 11 21:51:57.737872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.737848    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 21:51:57.750683 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.750651    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:57.781960 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.781928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.781991    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6jv\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782142 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.782325 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.782253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.873162 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.873127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:51:57.882746 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.882881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882767    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.882881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.882881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.882988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883046    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6jv\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883317 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883414 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883542 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.883845 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.883826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.885804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.885783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.886002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.885980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.890828 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.890807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6jv\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:57.890922 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:57.890857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token\") pod \"e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") " pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:58.039109 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.039064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:58.318441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.318414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 21:51:58.320906 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:51:58.320870    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bfa079_a16b_4b05_b52d_ffdeac8cca03.slice/crio-344b7f8b5d4ddfd5cdbd2c3a8321b48f86d075a3ff2adf4a5452b58cde3889bd WatchSource:0}: Error finding container 344b7f8b5d4ddfd5cdbd2c3a8321b48f86d075a3ff2adf4a5452b58cde3889bd: Status 404 returned error can't find the container with id 344b7f8b5d4ddfd5cdbd2c3a8321b48f86d075a3ff2adf4a5452b58cde3889bd
Sep 11 21:51:58.323469 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.323436    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:58.323554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.323507    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:58.323554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.323538    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:51:58.863797 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.863761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerStarted","Data":"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"}
Sep 11 21:51:58.864188 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.863821    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerStarted","Data":"344b7f8b5d4ddfd5cdbd2c3a8321b48f86d075a3ff2adf4a5452b58cde3889bd"}
Sep 11 21:51:58.883545 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:58.883481    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podStartSLOduration=1.8834610280000001 podStartE2EDuration="1.883461028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:51:58.882185148 +0000 UTC m=+1284.845209819" watchObservedRunningTime="2026-09-11 21:51:58.883461028 +0000 UTC m=+1284.846485677"
Sep 11 21:51:59.039783 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:59.039737    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:59.045224 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:59.045191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:59.868952 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:59.868916    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:51:59.870214 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:51:59.870191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:52:00.109000 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:00.108963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:00.808258 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:00.808224    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:00.903879 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:00.903845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:01.013290 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:01.013248    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:01.792411 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:01.792365    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:02.890287 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:02.890243    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:03.595450 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:03.595415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:04.298480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:04.298440    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:05.305707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:05.305670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:05.371612 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:05.371576    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:05.521064 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:05.521027    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:05.887080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:05.887044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:05.977384 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:05.977336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:06.000551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.000514    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 21:52:06.004869 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.004845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.007688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.007662    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7d7821-dockercfg-fntz2\""
Sep 11 21:52:06.007903 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.007862    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7d7821\""
Sep 11 21:52:06.013643 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.013616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 21:52:06.069189 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.069148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.069189 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.069190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dg2g\" (UniqueName: \"kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.170701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.170611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.170701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.170651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dg2g\" (UniqueName: \"kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.171333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.171311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.179547 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.179527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dg2g\" (UniqueName: \"kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g\") pod \"payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.318563 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.318527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.597787 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.597764    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 21:52:06.600075 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:52:06.600047    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb559bf1_c1a7_40d7_beba_6cbe1931e4bb.slice/crio-1e0338fd36fbcc20bffe36ee6f04f0d1fd6a04357b54e2bba3dd1c766e487dcb WatchSource:0}: Error finding container 1e0338fd36fbcc20bffe36ee6f04f0d1fd6a04357b54e2bba3dd1c766e487dcb: Status 404 returned error can't find the container with id 1e0338fd36fbcc20bffe36ee6f04f0d1fd6a04357b54e2bba3dd1c766e487dcb
Sep 11 21:52:06.911231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.911141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" event={"ID":"db559bf1-c1a7-40d7-beba-6cbe1931e4bb","Type":"ContainerStarted","Data":"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"}
Sep 11 21:52:06.911231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.911198    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" event={"ID":"db559bf1-c1a7-40d7-beba-6cbe1931e4bb","Type":"ContainerStarted","Data":"1e0338fd36fbcc20bffe36ee6f04f0d1fd6a04357b54e2bba3dd1c766e487dcb"}
Sep 11 21:52:06.911231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.911231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:06.926491 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:06.926422    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" podStartSLOduration=1.926401907 podStartE2EDuration="1.926401907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:06.925934247 +0000 UTC m=+1292.888958911" watchObservedRunningTime="2026-09-11 21:52:06.926401907 +0000 UTC m=+1292.889426556"
Sep 11 21:52:13.780583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:13.780543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:17.918628 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:17.918592    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 21:52:20.418592 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:20.418551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:23.665765 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:23.665729    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:52:27.349576 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:27.349537    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:28.106421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:28.106380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:28.772124 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:28.772086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:29.418631 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:29.418596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:30.109931 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:30.109897    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:30.878683 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:30.878643    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:30.980106 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:30.980069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:31.228374 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.228336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:31.529241 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.529158    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 21:52:31.534415 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.534389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.536465 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.536440    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4e0323-openshift-default-dockercfg-vph6f\""
Sep 11 21:52:31.545101 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.545074    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 21:52:31.562472 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.562438    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:31.633093 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633402 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.633552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.633513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtn4\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.695331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.695290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:31.734061 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtn4\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734617    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734694    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.734918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734899    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.735029 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.734989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.736698 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.736675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.737083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.737061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.741521 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.741502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.742596 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.742579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtn4\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4\") pod \"e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") " pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:31.847319 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:31.847227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:32.003328 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:32.003196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:32.149232 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:32.149206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 21:52:32.150950 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:52:32.150924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001fbf53_831b_4603_9fb3_2f2c3d608c53.slice/crio-cd08edb6ff9a4d59017ee7c81f8bb54f9671ac19efaa5efe2aa790ce5e8f1238 WatchSource:0}: Error finding container cd08edb6ff9a4d59017ee7c81f8bb54f9671ac19efaa5efe2aa790ce5e8f1238: Status 404 returned error can't find the container with id cd08edb6ff9a4d59017ee7c81f8bb54f9671ac19efaa5efe2aa790ce5e8f1238
Sep 11 21:52:32.153281 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:32.153247    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:32.153368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:32.153321    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:32.153368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:32.153351    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:33.052066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:33.052025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerStarted","Data":"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"}
Sep 11 21:52:33.052528 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:33.052087    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerStarted","Data":"cd08edb6ff9a4d59017ee7c81f8bb54f9671ac19efaa5efe2aa790ce5e8f1238"}
Sep 11 21:52:33.072102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:33.072041    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podStartSLOduration=2.072021021 podStartE2EDuration="2.072021021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:33.069827392 +0000 UTC m=+1319.032852041" watchObservedRunningTime="2026-09-11 21:52:33.072021021 +0000 UTC m=+1319.035045669"
Sep 11 21:52:33.847753 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:33.847689    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:33.853448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:33.853415    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:34.057640 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:34.057604    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:34.058552 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:34.058532    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:52:34.309158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:34.309114    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:34.770632 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:34.770581    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:34.980390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:34.980356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:35.080568 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:35.080531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:35.902004 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:35.901974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:36.070702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.070671    2582 generic.go:356] "Generic (PLEG): container finished" podID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerID="ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482" exitCode=0
Sep 11 21:52:36.070905 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.070747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerDied","Data":"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"}
Sep 11 21:52:36.071271 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.071245    2582 scope.go:117] "RemoveContainer" containerID="ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"
Sep 11 21:52:36.072357 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.072330    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:36.072421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.072398    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:36.072459 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.072431    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:52:36.765540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:36.765502    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:37.077769 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:37.077664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerStarted","Data":"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"}
Sep 11 21:52:37.671036 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:37.670999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:38.039306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:38.039268    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:52:38.039701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:38.039490    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 21:52:38.039701 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:38.039549    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 21:52:38.329314 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:38.329218    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:39.039968 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:39.039932    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 21:52:39.040370 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:39.040006    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 21:52:39.109290 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:39.109256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:39.857959 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:39.857922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:39.959079 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:39.959042    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:40.014916 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:40.014888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:40.040073 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:40.040038    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 21:52:40.040428 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:40.040091    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 21:52:40.320087 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:40.320056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:40.405742 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:40.405157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:41.040102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:41.040060    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 21:52:41.040499 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:41.040117    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 21:52:42.040540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:42.040502    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 21:52:42.041018 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:42.040582    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 21:52:43.043665 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:43.043631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:52:43.044060 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:43.043969    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:52:43.044652 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:43.044631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 21:52:43.379520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:43.379429    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:44.196067 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:44.196036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:44.885151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:44.885110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:45.607936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:45.607896    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:46.280524 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:46.280492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:47.074755 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:47.074704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:47.174173 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:47.174139    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:47.746702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:47.746664    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:48.564426 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:48.564387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:49.427480 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:49.427442    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:49.593847 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:49.593806    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:52:49.725260 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:49.725157    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:50.207087 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:50.207052    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:51.022635 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:51.022591    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:51.804177 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:51.804142    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:52.460234 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:52.460183    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:53.179763 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:53.179702    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:54.097587 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:54.097518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:52:54.769494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:52:54.769460    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.359104 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:00.359070    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.458378 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:00.458326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.554463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:00.554423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:00.614959 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:00.614869    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:01.049219 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.049169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:01.233809 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.233776    2582 generic.go:356] "Generic (PLEG): container finished" podID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerID="e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9" exitCode=0
Sep 11 21:53:01.234006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.233856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerDied","Data":"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"}
Sep 11 21:53:01.234353 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.234335    2582 scope.go:117] "RemoveContainer" containerID="e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"
Sep 11 21:53:01.235564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.235537    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:53:01.235621 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.235603    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:53:01.235659 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.235636    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:53:01.790674 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:01.790573    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.241386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:02.241350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerStarted","Data":"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"}
Sep 11 21:53:02.708121 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:02.707999    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:02.847873 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:02.847839    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:53:02.849251 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:02.849222    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:53:02.849380 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:02.849271    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 21:53:03.457903 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:03.457871    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:03.785664 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:03.785631    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:03.848175 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:03.848141    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:53:03.848548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:03.848193    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 21:53:04.057601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:04.057501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:53:04.183943 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:04.183908    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:04.824547 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:04.824509    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:04.848139 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:04.848106    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:53:04.848329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:04.848162    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 21:53:05.472931 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:05.472886    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:05.848418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:05.848374    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:53:05.848838 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:05.848441    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 21:53:06.289421 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:06.289384    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:06.848164 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:06.848126    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 11 21:53:06.848351 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:06.848199    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 11 21:53:07.852375 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:07.852343    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:53:07.853442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:07.853419    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 21:53:08.763295 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:08.763259    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:53:08.810422 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:08.810386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:23.784308 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:23.784250    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" podUID="edf881d9-797c-47f0-8894-5e265e024477" containerName="main" containerID="cri-o://adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346" gracePeriod=45
Sep 11 21:53:23.836572 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:23.836535    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" podUID="edf881d9-797c-47f0-8894-5e265e024477" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": dial tcp 10.132.0.56:8000: connect: connection refused"
Sep 11 21:53:24.046938 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.046862    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:53:24.091902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.091861    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.091929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.091979    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltjlx\" (UniqueName: \"kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092047    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs\") pod \"edf881d9-797c-47f0-8894-5e265e024477\" (UID: \"edf881d9-797c-47f0-8894-5e265e024477\") "
Sep 11 21:53:24.092302 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home" (OuterVolumeSpecName: "home") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:24.092466 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092442    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:24.092616 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092559    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-home\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.092616 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092563    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache" (OuterVolumeSpecName: "model-cache") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:24.092616 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.092580    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-tmp-dir\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.095184 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.095150    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx" (OuterVolumeSpecName: "kube-api-access-ltjlx") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "kube-api-access-ltjlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:24.095184 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.095172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:53:24.095314 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.095174    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm" (OuterVolumeSpecName: "dshm") pod "edf881d9-797c-47f0-8894-5e265e024477" (UID: "edf881d9-797c-47f0-8894-5e265e024477"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:24.193231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.193201    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf881d9-797c-47f0-8894-5e265e024477-tls-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.193231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.193228    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-model-cache\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.193231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.193239    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltjlx\" (UniqueName: \"kubernetes.io/projected/edf881d9-797c-47f0-8894-5e265e024477-kube-api-access-ltjlx\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.193479 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.193248    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf881d9-797c-47f0-8894-5e265e024477-dshm\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:24.369254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.369154    2582 generic.go:356] "Generic (PLEG): container finished" podID="edf881d9-797c-47f0-8894-5e265e024477" containerID="adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346" exitCode=0
Sep 11 21:53:24.369254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.369194    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"
Sep 11 21:53:24.369254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.369243    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" event={"ID":"edf881d9-797c-47f0-8894-5e265e024477","Type":"ContainerDied","Data":"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"}
Sep 11 21:53:24.369481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.369283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf" event={"ID":"edf881d9-797c-47f0-8894-5e265e024477","Type":"ContainerDied","Data":"b7711cc3b687ab3218785fe2eb6a4499b3ab889d6cf2b24ba79c7e037fcd9295"}
Sep 11 21:53:24.369481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.369304    2582 scope.go:117] "RemoveContainer" containerID="adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"
Sep 11 21:53:24.378666 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.378649    2582 scope.go:117] "RemoveContainer" containerID="adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"
Sep 11 21:53:24.378980 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:53:24.378957    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346\": container with ID starting with adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346 not found: ID does not exist" containerID="adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"
Sep 11 21:53:24.379066 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.378991    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346"} err="failed to get container status \"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346\": rpc error: code = NotFound desc = could not find container \"adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346\": container with ID starting with adf2337ed187959a453c61e0f09e7c5abd8b54282a5295858b071923408a8346 not found: ID does not exist"
Sep 11 21:53:24.389638 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.389612    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:53:24.393201 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.393179    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9f68da5e/e2e-lifecycle-model-9f68da5e-kserve-6c459d8459-sszsf"]
Sep 11 21:53:24.456460 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:24.456424    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf881d9-797c-47f0-8894-5e265e024477" path="/var/lib/kubelet/pods/edf881d9-797c-47f0-8894-5e265e024477/volumes"
Sep 11 21:53:26.559355 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:26.559322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:53:26.785458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:26.785426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:45.082999 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:45.082965    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:45.326583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:45.326541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:46.113611 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:46.113579    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:51.531972 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:51.531933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:51.608359 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:51.608321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:51.772436 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:51.772402    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:53:51.772647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:51.772626    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" containerID="cri-o://4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f" gracePeriod=30
Sep 11 21:53:51.795799 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:51.795683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:52.061194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:52.061100    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:53:53.042742 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:53.042678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.141937 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:53.141901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.242842 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:53.242799    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.842292 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:53.842216    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:53.936455 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:53.936422    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:54.037333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:54.037295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:55.047215 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:55.047180    2582 patch_prober.go:28] interesting pod/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:53:55.047618 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:55.047243    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:53:55.294939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:55.294902    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:53:57.031741 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.031697    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:53:57.118445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118424    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118567    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118624    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118747 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:57.118915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118730    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-458vm\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118805    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:57.118915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.118915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.118878    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket\") pod \"70c0d014-7b38-42e3-b5d7-4a34264687c0\" (UID: \"70c0d014-7b38-42e3-b5d7-4a34264687c0\") "
Sep 11 21:53:57.119180 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119154    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:57.119270 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:53:57.119495 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119276    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:57.119773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119698    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.119773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119755    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.119773 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119771    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.120026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119785    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c0d014-7b38-42e3-b5d7-4a34264687c0-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.120026 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.119799    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.121149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.121123    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:53:57.121149 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.121136    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:57.121451 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.121425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:53:57.121579 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.121555    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm" (OuterVolumeSpecName: "kube-api-access-458vm") pod "70c0d014-7b38-42e3-b5d7-4a34264687c0" (UID: "70c0d014-7b38-42e3-b5d7-4a34264687c0"). InnerVolumeSpecName "kube-api-access-458vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:53:57.220798 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.220764    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.220798 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.220792    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.220798 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.220803    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c0d014-7b38-42e3-b5d7-4a34264687c0-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.221049 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.220812    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458vm\" (UniqueName: \"kubernetes.io/projected/70c0d014-7b38-42e3-b5d7-4a34264687c0-kube-api-access-458vm\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:53:57.543776 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.543738    2582 generic.go:356] "Generic (PLEG): container finished" podID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerID="4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f" exitCode=0
Sep 11 21:53:57.543949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.543789    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"
Sep 11 21:53:57.543949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.543813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerDied","Data":"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"}
Sep 11 21:53:57.543949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.543875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7" event={"ID":"70c0d014-7b38-42e3-b5d7-4a34264687c0","Type":"ContainerDied","Data":"476ddb0e91f455e1f563fc2cf4742dd9eca6e6a0bf826eb2e1d8dc1c70005c8f"}
Sep 11 21:53:57.543949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.543899    2582 scope.go:117] "RemoveContainer" containerID="4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"
Sep 11 21:53:57.554198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.554177    2582 scope.go:117] "RemoveContainer" containerID="90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"
Sep 11 21:53:57.563220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.563202    2582 scope.go:117] "RemoveContainer" containerID="4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"
Sep 11 21:53:57.563549 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:53:57.563518    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f\": container with ID starting with 4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f not found: ID does not exist" containerID="4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"
Sep 11 21:53:57.563754 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.563562    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f"} err="failed to get container status \"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f\": rpc error: code = NotFound desc = could not find container \"4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f\": container with ID starting with 4707a2f432d9a9d2dc71163c62b9d1bc8293219617afa7e1f23b1c86a0961e9f not found: ID does not exist"
Sep 11 21:53:57.563754 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.563656    2582 scope.go:117] "RemoveContainer" containerID="90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"
Sep 11 21:53:57.564150 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:53:57.564062    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a\": container with ID starting with 90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a not found: ID does not exist" containerID="90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"
Sep 11 21:53:57.564150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.564095    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a"} err="failed to get container status \"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a\": rpc error: code = NotFound desc = could not find container \"90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a\": container with ID starting with 90c8b5d4914c68e9d20b64616fed4a6ed5d0c64f97e45a28f454589fffe2c36a not found: ID does not exist"
Sep 11 21:53:57.566296 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.566275    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:53:57.570806 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:57.570784    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9f68da5e-openshift-default-76d446675b-jp7t7"]
Sep 11 21:53:58.455833 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:53:58.455791    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" path="/var/lib/kubelet/pods/70c0d014-7b38-42e3-b5d7-4a34264687c0/volumes"
Sep 11 21:54:01.270847 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:01.270801    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:01.335258 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:01.335225    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:01.957152 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:01.957119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:07.767834 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:07.767800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:07.833033 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:07.832995    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:07.918386 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:07.918345    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:08.064441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:08.064349    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:08.201420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:08.201380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:08.523841 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:08.523802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:08.624102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:08.624047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:11.978677 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:11.978634    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:18.477679 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:18.477639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:18.691151 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:18.691119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:23.463085 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463043    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:54:23.463505 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463491    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf881d9-797c-47f0-8894-5e265e024477" containerName="main"
Sep 11 21:54:23.463551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463507    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf881d9-797c-47f0-8894-5e265e024477" containerName="main"
Sep 11 21:54:23.463551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463527    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:23.463551 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463536    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:23.463688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463551    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:23.463688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463557    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:23.463688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463620    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:23.463688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.463632    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf881d9-797c-47f0-8894-5e265e024477" containerName="main"
Sep 11 21:54:23.467001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.466983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.469288 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.469265    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"openshift-service-ca.crt\""
Sep 11 21:54:23.469413 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.469313    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"default-dockercfg-lp4bk\""
Sep 11 21:54:23.469902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.469880    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"kube-root-ca.crt\""
Sep 11 21:54:23.470018 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.469927    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4e0323\"/\"auth-test-model-4e0323-kserve-self-signed-certs\""
Sep 11 21:54:23.475622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.475603    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:54:23.553473 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.553421    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:23.584705 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.584921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.584921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd6r2\" (UniqueName: \"kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.584921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584839    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.585059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.585059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.584941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686164 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd6r2\" (UniqueName: \"kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686762 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.686892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.686804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.688706 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.688688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.688921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.688901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.693965 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.693941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd6r2\" (UniqueName: \"kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2\") pod \"auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") " pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:23.762345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.762309    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:54:23.777515 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:23.777490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:24.051695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:24.051617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:54:24.054377 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:54:24.054348    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d11922b_fd01_423f_872c_2fd21b8018a2.slice/crio-ca23d8860f9ebf5c55a087f415d09f0ad92f7e82a7c140be11046dfde84c0b92 WatchSource:0}: Error finding container ca23d8860f9ebf5c55a087f415d09f0ad92f7e82a7c140be11046dfde84c0b92: Status 404 returned error can't find the container with id ca23d8860f9ebf5c55a087f415d09f0ad92f7e82a7c140be11046dfde84c0b92
Sep 11 21:54:24.689486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:24.689445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" event={"ID":"5d11922b-fd01-423f-872c-2fd21b8018a2","Type":"ContainerStarted","Data":"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"}
Sep 11 21:54:24.689486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:24.689503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" event={"ID":"5d11922b-fd01-423f-872c-2fd21b8018a2","Type":"ContainerStarted","Data":"ca23d8860f9ebf5c55a087f415d09f0ad92f7e82a7c140be11046dfde84c0b92"}
Sep 11 21:54:24.706014 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:24.705965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" podStartSLOduration=1.705949352 podStartE2EDuration="1.705949352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:24.70363931 +0000 UTC m=+1430.666663959" watchObservedRunningTime="2026-09-11 21:54:24.705949352 +0000 UTC m=+1430.668974029"
Sep 11 21:54:33.778542 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:33.778444    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:33.778542 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:33.778505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:33.791843 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:33.791809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:34.755018 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:34.754985    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:54:39.645588 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:39.645531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:40.935396 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:40.935352    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:42.765210 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:42.765171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:45.556084 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:45.556047    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:45.612062 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:45.612023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:45.896537 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:45.896448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:46.180836 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.180763    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:54:46.181265 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.181250    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="70c0d014-7b38-42e3-b5d7-4a34264687c0" containerName="istio-proxy"
Sep 11 21:54:46.185906 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.185879    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.188041 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.188021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1132e5-openshift-default-dockercfg-mngl7\""
Sep 11 21:54:46.206861 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.206831    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:54:46.210122 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.210094    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:46.298989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.298947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.298989 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.298984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299225 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299225 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299225 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mf7\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.299463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.299349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.311005 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.310966    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:46.400187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400187 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400456 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400396    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400760 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mf7\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.400760 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.401235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.400963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.401235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.401003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.401235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.401102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.401235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.401184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.401235 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.401233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.402578 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.402560    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.403086 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.403068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.408042 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.408020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mf7\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.408153 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.408093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token\") pod \"e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") " pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.497298 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.497258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:46.761165 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.761086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:46.772520 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.772497    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:54:46.774699 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:54:46.774667    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440ab0fd_d73f_480d_8702_5d7a90778ca4.slice/crio-4b8981fa0be83c0a1765e46f95c5ea4a741f5565e0791f8ad744008f14ea098b WatchSource:0}: Error finding container 4b8981fa0be83c0a1765e46f95c5ea4a741f5565e0791f8ad744008f14ea098b: Status 404 returned error can't find the container with id 4b8981fa0be83c0a1765e46f95c5ea4a741f5565e0791f8ad744008f14ea098b
Sep 11 21:54:46.777019 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.776984    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:54:46.777120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.777066    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:54:46.777120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.777113    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:54:46.805617 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:46.805591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerStarted","Data":"4b8981fa0be83c0a1765e46f95c5ea4a741f5565e0791f8ad744008f14ea098b"}
Sep 11 21:54:47.812291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:47.812257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerStarted","Data":"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"}
Sep 11 21:54:47.832196 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:47.832139    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podStartSLOduration=1.832119378 podStartE2EDuration="1.832119378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:47.829573378 +0000 UTC m=+1453.792598031" watchObservedRunningTime="2026-09-11 21:54:47.832119378 +0000 UTC m=+1453.795144026"
Sep 11 21:54:48.498240 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:48.498203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:48.503730 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:48.503694    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:48.818342 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:48.818247    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:48.819437 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:48.819414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:54:49.060684 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:49.060613    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:49.731358 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:49.731327    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:49.821810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:49.821772    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:50.715540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:50.715508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:51.489397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:51.489367    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:52.393861 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:52.393829    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:52.762127 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:52.762083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:54:53.020646 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:53.020551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:53.771519 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:53.771480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:54.564470 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.564439    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:54:54.568511 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.568487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.570779 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.570751    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1132e5-serving-cert\""
Sep 11 21:54:54.579404 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.579380    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:54:54.596383 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.596358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:54.687831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.687799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fr2v\" (UniqueName: \"kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.688020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.687872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.688020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.687893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.688020 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.687914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.713342 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.713304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:54.789252 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.789215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fr2v\" (UniqueName: \"kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.789764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.789321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.789764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.789355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.789764 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:54:54.789456    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-serving-cert: secret "maas-api-e2e-api-a-1132e5-serving-cert" not found
Sep 11 21:54:54.789764 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.789483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.789764 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:54:54.789555    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls podName:bea45ee7-a958-4b79-8599-19bbc3e02889 nodeName:}" failed. No retries permitted until 2026-09-11 21:54:55.289525612 +0000 UTC m=+1461.252550243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls") pod "maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" (UID: "bea45ee7-a958-4b79-8599-19bbc3e02889") : secret "maas-api-e2e-api-a-1132e5-serving-cert" not found
Sep 11 21:54:54.790158 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.790137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.791999 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.791967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.808425 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.808396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fr2v\" (UniqueName: \"kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:54.956129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:54.956023    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:55.083104 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.083064    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:55.181619 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.181582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:54:55.190262 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.190230    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:54:55.194012 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.193995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.196702 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.196678    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1132e5\""
Sep 11 21:54:55.196838 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.196787    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1132e5-dockercfg-8q95d\""
Sep 11 21:54:55.207154 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.207098    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:54:55.296006 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.295962    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.296194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.296109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxg8m\" (UniqueName: \"kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.296194 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.296169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:55.299029 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.298999    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") pod \"maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:55.396946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.396903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.397168 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.397009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxg8m\" (UniqueName: \"kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.397678 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.397649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.405486 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.405465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxg8m\" (UniqueName: \"kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m\") pod \"payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.483744 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.483680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:55.505102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.505072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:55.760083 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.760054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:54:55.762411 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:54:55.762380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea45ee7_a958_4b79_8599_19bbc3e02889.slice/crio-cafa541cb1aabaaabc08566b1fae17aa977f115d27c66c389bf10839426116c5 WatchSource:0}: Error finding container cafa541cb1aabaaabc08566b1fae17aa977f115d27c66c389bf10839426116c5: Status 404 returned error can't find the container with id cafa541cb1aabaaabc08566b1fae17aa977f115d27c66c389bf10839426116c5
Sep 11 21:54:55.763808 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.763791    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:54:55.791759 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.791735    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:54:55.794041 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:54:55.794015    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2520c489_1e77_4bb3_8661_d3b3c8cad2c8.slice/crio-4cc6414ec7460ed17ec2baffe3e9151a63a72b1a25c286b0badf1e5e557a2bc5 WatchSource:0}: Error finding container 4cc6414ec7460ed17ec2baffe3e9151a63a72b1a25c286b0badf1e5e557a2bc5: Status 404 returned error can't find the container with id 4cc6414ec7460ed17ec2baffe3e9151a63a72b1a25c286b0badf1e5e557a2bc5
Sep 11 21:54:55.856940 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.856909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" event={"ID":"2520c489-1e77-4bb3-8661-d3b3c8cad2c8","Type":"ContainerStarted","Data":"4cc6414ec7460ed17ec2baffe3e9151a63a72b1a25c286b0badf1e5e557a2bc5"}
Sep 11 21:54:55.858161 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:55.858138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" event={"ID":"bea45ee7-a958-4b79-8599-19bbc3e02889","Type":"ContainerStarted","Data":"cafa541cb1aabaaabc08566b1fae17aa977f115d27c66c389bf10839426116c5"}
Sep 11 21:54:56.865238 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.865192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" event={"ID":"2520c489-1e77-4bb3-8661-d3b3c8cad2c8","Type":"ContainerStarted","Data":"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"}
Sep 11 21:54:56.865709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.865260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:54:56.866838 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.866809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" event={"ID":"bea45ee7-a958-4b79-8599-19bbc3e02889","Type":"ContainerStarted","Data":"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"}
Sep 11 21:54:56.866941 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.866867    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:54:56.881333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.881285    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" podStartSLOduration=1.881270254 podStartE2EDuration="1.881270254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:56.879854333 +0000 UTC m=+1462.842878994" watchObservedRunningTime="2026-09-11 21:54:56.881270254 +0000 UTC m=+1462.844294934"
Sep 11 21:54:56.895901 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:54:56.895854    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" podStartSLOduration=2.48648569 podStartE2EDuration="2.895840269s" totalImagesPullingTime="409.354579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:54:56.894012251 +0000 UTC m=+1462.857036901" watchObservedRunningTime="2026-09-11 21:54:56.895840269 +0000 UTC m=+1462.858864919"
Sep 11 21:55:02.877793 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:02.877757    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:55:07.873781 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:07.873741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:55:16.505683 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:16.505640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:17.479598 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:17.479545    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:18.289483 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:18.289436    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:19.021300 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:19.021269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:19.725539 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:19.725498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:19.845933 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:19.845893    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:20.291588 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:20.291547    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:20.607135 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:20.607037    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:20.709990 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:20.709930    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:21.135389 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:21.135354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:21.413750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:21.413639    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:22.058500 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:22.058463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:22.991033 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:22.990993    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:23.645910 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:23.645866    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:23.744133 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:23.744098    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:24.399929 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:24.399885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:25.042693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:25.042658    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:25.724830 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:25.724793    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:26.689915 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:26.689885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:27.040442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.040403    2582 generic.go:356] "Generic (PLEG): container finished" podID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerID="071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f" exitCode=0
Sep 11 21:55:27.040442 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.040444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerDied","Data":"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"}
Sep 11 21:55:27.040893 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.040883    2582 scope.go:117] "RemoveContainer" containerID="071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"
Sep 11 21:55:27.042060 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.042031    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:55:27.042128 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.042104    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:55:27.042167 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.042135    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 21:55:27.403694 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:27.403605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:28.037981 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.037941    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:28.048195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.048168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerStarted","Data":"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"}
Sep 11 21:55:28.503133 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.502930    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:55:28.503458 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.503433    2582 patch_prober.go:28] interesting pod/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 21:55:28.503646 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.503622    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 21:55:28.910810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:28.897503    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:29.000741 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.000554    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:29.349095 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.349059    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:29.448881 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.448845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:29.456462 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.456434    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:55:29.459183 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.459166    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.461297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.461277    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-24efea\""
Sep 11 21:55:29.461405 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.461299    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-24efea-dockercfg-dl8s4\""
Sep 11 21:55:29.476306 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.476281    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:55:29.512200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.512163    2582 patch_prober.go:28] interesting pod/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 21:55:29.512493 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.512460    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 21:55:29.517034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.517009    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:55:29.520648 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.520623    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:55:29.520803 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.520771    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.543785 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.543754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.543785 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.543793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5gh\" (UniqueName: \"kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.645129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.645026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hl5\" (UniqueName: \"kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.645129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.645097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.645385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.645148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.645385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.645181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5gh\" (UniqueName: \"kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.645869 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.645849    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.654834 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.654811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5gh\" (UniqueName: \"kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh\") pod \"payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.745979 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.745943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hl5\" (UniqueName: \"kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.746162 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.746000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.746691 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.746669    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.753341 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.753313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hl5\" (UniqueName: \"kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5\") pod \"payload-processing-e2e-api-b-24efea-7d5d9d647-fls48\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") " pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:29.770815 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.770790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:29.832613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:29.832525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:30.072275 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:30.072251    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:55:30.074034 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:55:30.074006    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672f2dd8_7b11_4020_9209_0fbaf891b2ba.slice/crio-a304eb369f7d1a7a6481713d4e67fe738d0f7f24b626171fb154ff9539e724fc WatchSource:0}: Error finding container a304eb369f7d1a7a6481713d4e67fe738d0f7f24b626171fb154ff9539e724fc: Status 404 returned error can't find the container with id a304eb369f7d1a7a6481713d4e67fe738d0f7f24b626171fb154ff9539e724fc
Sep 11 21:55:30.140756 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:30.140709    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:55:30.151842 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:55:30.151815    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0ec857_4b5e_4c93_99ee_8c7001c7b97c.slice/crio-94eb1243066ecb008d5c3e2944e7aff57c8c986aa3bd1f5650fedbe115aee44c WatchSource:0}: Error finding container 94eb1243066ecb008d5c3e2944e7aff57c8c986aa3bd1f5650fedbe115aee44c: Status 404 returned error can't find the container with id 94eb1243066ecb008d5c3e2944e7aff57c8c986aa3bd1f5650fedbe115aee44c
Sep 11 21:55:30.497892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:30.497847    2582 patch_prober.go:28] interesting pod/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 21:55:30.498408 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:30.497926    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 21:55:31.068326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.068288    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" event={"ID":"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c","Type":"ContainerStarted","Data":"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"}
Sep 11 21:55:31.068527 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.068350    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" event={"ID":"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c","Type":"ContainerStarted","Data":"94eb1243066ecb008d5c3e2944e7aff57c8c986aa3bd1f5650fedbe115aee44c"}
Sep 11 21:55:31.068527 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.068485    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:31.070007 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.069982    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" event={"ID":"672f2dd8-7b11-4020-9209-0fbaf891b2ba","Type":"ContainerStarted","Data":"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"}
Sep 11 21:55:31.070144 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.070032    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:31.070144 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.070055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" event={"ID":"672f2dd8-7b11-4020-9209-0fbaf891b2ba","Type":"ContainerStarted","Data":"a304eb369f7d1a7a6481713d4e67fe738d0f7f24b626171fb154ff9539e724fc"}
Sep 11 21:55:31.085463 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.085402    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" podStartSLOduration=2.085382751 podStartE2EDuration="2.085382751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:31.082230148 +0000 UTC m=+1497.045254798" watchObservedRunningTime="2026-09-11 21:55:31.085382751 +0000 UTC m=+1497.048407400"
Sep 11 21:55:31.498341 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.498292    2582 patch_prober.go:28] interesting pod/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 21:55:31.498869 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:31.498376    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 21:55:32.498621 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:32.498582    2582 patch_prober.go:28] interesting pod/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 21:55:32.499248 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:32.499217    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 21:55:33.502348 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:33.502317    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:55:33.502751 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:33.502600    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:55:33.503369 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:33.503354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:55:33.522167 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:33.522119    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" podStartSLOduration=4.522103941 podStartE2EDuration="4.522103941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:31.101223167 +0000 UTC m=+1497.064247818" watchObservedRunningTime="2026-09-11 21:55:33.522103941 +0000 UTC m=+1499.485128592"
Sep 11 21:55:42.077111 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:42.077075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:55:42.077514 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:42.077245    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:55:50.908204 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:50.908148    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:51.534795 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:51.534753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:52.278792 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:52.278757    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:52.927606 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:52.927570    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:54.236826 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:54.236794    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:54.337192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:54.337150    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:54.437937 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:54.437901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:54.974756 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:54.974700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:55.633647 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:55.633558    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:56.401658 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:56.401624    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:57.163608 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:57.163574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:57.875135 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:57.875101    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:58.692561 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:58.692524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:59.349146 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:59.349106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:55:59.992815 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:55:59.992780    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:00.892058 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:00.892011    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:00.973041 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:00.972999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:56:01.587133 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:01.587088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:02.212598 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:02.212559    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:02.994423 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:02.994374    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:06.888096 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:06.888060    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.311630 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.311589    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.421861 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.421825    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:56:12.422148 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.422071    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" podUID="bea45ee7-a958-4b79-8599-19bbc3e02889" containerName="maas-api" containerID="cri-o://2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274" gracePeriod=30
Sep 11 21:56:12.451372 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.451337    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.488541 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.488504    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:56:12.488813 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.488785    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" podUID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" containerName="payload-pre-processing" containerID="cri-o://0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1" gracePeriod=30
Sep 11 21:56:12.552886 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.552847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.653904 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.653169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.712853 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.712813    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:12.786075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.786048    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:56:12.790685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.790653    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:56:12.883014 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.882920    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fr2v\" (UniqueName: \"kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v\") pod \"bea45ee7-a958-4b79-8599-19bbc3e02889\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") "
Sep 11 21:56:12.883014 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.882965    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume\") pod \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") "
Sep 11 21:56:12.883014 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca\") pod \"bea45ee7-a958-4b79-8599-19bbc3e02889\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") "
Sep 11 21:56:12.883301 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883137    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") pod \"bea45ee7-a958-4b79-8599-19bbc3e02889\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") "
Sep 11 21:56:12.883366 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883301    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxg8m\" (UniqueName: \"kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m\") pod \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\" (UID: \"2520c489-1e77-4bb3-8661-d3b3c8cad2c8\") "
Sep 11 21:56:12.883418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883363    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs\") pod \"bea45ee7-a958-4b79-8599-19bbc3e02889\" (UID: \"bea45ee7-a958-4b79-8599-19bbc3e02889\") "
Sep 11 21:56:12.883418 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883375    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bea45ee7-a958-4b79-8599-19bbc3e02889" (UID: "bea45ee7-a958-4b79-8599-19bbc3e02889"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:12.883527 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883417    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2520c489-1e77-4bb3-8661-d3b3c8cad2c8" (UID: "2520c489-1e77-4bb3-8661-d3b3c8cad2c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:12.883872 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883839    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:12.883992 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.883878    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bea45ee7-a958-4b79-8599-19bbc3e02889-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:12.886113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.886079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v" (OuterVolumeSpecName: "kube-api-access-8fr2v") pod "bea45ee7-a958-4b79-8599-19bbc3e02889" (UID: "bea45ee7-a958-4b79-8599-19bbc3e02889"). InnerVolumeSpecName "kube-api-access-8fr2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:12.886297 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.886271    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bea45ee7-a958-4b79-8599-19bbc3e02889" (UID: "bea45ee7-a958-4b79-8599-19bbc3e02889"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:12.886374 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.886310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m" (OuterVolumeSpecName: "kube-api-access-zxg8m") pod "2520c489-1e77-4bb3-8661-d3b3c8cad2c8" (UID: "2520c489-1e77-4bb3-8661-d3b3c8cad2c8"). InnerVolumeSpecName "kube-api-access-zxg8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:12.886446 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.886427    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bea45ee7-a958-4b79-8599-19bbc3e02889" (UID: "bea45ee7-a958-4b79-8599-19bbc3e02889"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:12.984971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.984931    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fr2v\" (UniqueName: \"kubernetes.io/projected/bea45ee7-a958-4b79-8599-19bbc3e02889-kube-api-access-8fr2v\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:12.984971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.984963    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:12.984971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.984973    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxg8m\" (UniqueName: \"kubernetes.io/projected/2520c489-1e77-4bb3-8661-d3b3c8cad2c8-kube-api-access-zxg8m\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:12.984971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:12.984982    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bea45ee7-a958-4b79-8599-19bbc3e02889-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:13.004002 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.003952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:13.106680 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.106650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:13.301180 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.301148    2582 generic.go:356] "Generic (PLEG): container finished" podID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" containerID="0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1" exitCode=0
Sep 11 21:56:13.301368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.301194    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"
Sep 11 21:56:13.301368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.301238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" event={"ID":"2520c489-1e77-4bb3-8661-d3b3c8cad2c8","Type":"ContainerDied","Data":"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"}
Sep 11 21:56:13.301368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.301289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s" event={"ID":"2520c489-1e77-4bb3-8661-d3b3c8cad2c8","Type":"ContainerDied","Data":"4cc6414ec7460ed17ec2baffe3e9151a63a72b1a25c286b0badf1e5e557a2bc5"}
Sep 11 21:56:13.301368 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.301319    2582 scope.go:117] "RemoveContainer" containerID="0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"
Sep 11 21:56:13.302870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.302785    2582 generic.go:356] "Generic (PLEG): container finished" podID="bea45ee7-a958-4b79-8599-19bbc3e02889" containerID="2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274" exitCode=0
Sep 11 21:56:13.302870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.302861    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"
Sep 11 21:56:13.302870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.302872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" event={"ID":"bea45ee7-a958-4b79-8599-19bbc3e02889","Type":"ContainerDied","Data":"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"}
Sep 11 21:56:13.302870 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.302915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt" event={"ID":"bea45ee7-a958-4b79-8599-19bbc3e02889","Type":"ContainerDied","Data":"cafa541cb1aabaaabc08566b1fae17aa977f115d27c66c389bf10839426116c5"}
Sep 11 21:56:13.313567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.312892    2582 scope.go:117] "RemoveContainer" containerID="0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"
Sep 11 21:56:13.313567 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:13.313286    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1\": container with ID starting with 0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1 not found: ID does not exist" containerID="0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"
Sep 11 21:56:13.313567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.313323    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1"} err="failed to get container status \"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1\": rpc error: code = NotFound desc = could not find container \"0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1\": container with ID starting with 0e66370254dc3e45112e3d60a52378d0956e207c19778b659430d9a18a1567c1 not found: ID does not exist"
Sep 11 21:56:13.313567 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.313358    2582 scope.go:117] "RemoveContainer" containerID="2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"
Sep 11 21:56:13.325015 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.324997    2582 scope.go:117] "RemoveContainer" containerID="2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"
Sep 11 21:56:13.325642 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:13.325572    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274\": container with ID starting with 2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274 not found: ID does not exist" containerID="2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"
Sep 11 21:56:13.325642 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.325616    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274"} err="failed to get container status \"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274\": rpc error: code = NotFound desc = could not find container \"2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274\": container with ID starting with 2257c4f482d694f324118adc99a7203b829546aaf16847af8158962272d18274 not found: ID does not exist"
Sep 11 21:56:13.328044 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.328025    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:56:13.332200 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.332180    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1132e5-8d667bcbb-gnk5s"]
Sep 11 21:56:13.342333 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.342310    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:56:13.344397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.344376    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1132e5-6df8b78544-rtbrt"]
Sep 11 21:56:13.772179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:13.772133    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:14.455710 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:14.455676    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" path="/var/lib/kubelet/pods/2520c489-1e77-4bb3-8661-d3b3c8cad2c8/volumes"
Sep 11 21:56:14.456080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:14.456065    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea45ee7-a958-4b79-8599-19bbc3e02889" path="/var/lib/kubelet/pods/bea45ee7-a958-4b79-8599-19bbc3e02889/volumes"
Sep 11 21:56:14.559947 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:14.559912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:15.194205 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:15.194170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:15.831433 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:15.831399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:16.698568 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:16.698526    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:17.381646 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:17.381607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:18.159507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:18.159480    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:19.236088 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:19.236056    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:19.293611 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:19.293575    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:19.470848 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:19.470815    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:56:19.471195 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:19.471165    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy" containerID="cri-o://91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498" gracePeriod=30
Sep 11 21:56:19.501144 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:19.501033    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:20.071863 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:20.071818    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:20.795254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:20.795215    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:21.684339 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:21.684306    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:22.367896 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:22.367851    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:23.012114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:23.012079    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:23.823935 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:23.823883    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:24.584080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.584049    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:24.742574 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.742513    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:56:24.913303 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913257    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913303 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913378    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913479    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913513    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8mf7\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7\") pod \"440ab0fd-d73f-480d-8702-5d7a90778ca4\" (UID: \"440ab0fd-d73f-480d-8702-5d7a90778ca4\") "
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913805    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:24.913852 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913849    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913885    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data" (OuterVolumeSpecName: "istio-data") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.913993    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.914150    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.914164    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.914173    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.914181    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.914228 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.914190    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab0fd-d73f-480d-8702-5d7a90778ca4-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:24.915688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.915665    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token" (OuterVolumeSpecName: "istio-token") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:24.915889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.915871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 21:56:24.916080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.916058    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:56:24.916209 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:24.916181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7" (OuterVolumeSpecName: "kube-api-access-x8mf7") pod "440ab0fd-d73f-480d-8702-5d7a90778ca4" (UID: "440ab0fd-d73f-480d-8702-5d7a90778ca4"). InnerVolumeSpecName "kube-api-access-x8mf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:25.015446 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.015330    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8mf7\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-kube-api-access-x8mf7\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:25.015446 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.015381    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:25.015446 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.015395    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:25.015446 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.015405    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab0fd-d73f-480d-8702-5d7a90778ca4-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:25.256669 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.256638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:25.371930 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.371846    2582 generic.go:356] "Generic (PLEG): container finished" podID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerID="91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498" exitCode=0
Sep 11 21:56:25.371930 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.371895    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"
Sep 11 21:56:25.372120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.371927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerDied","Data":"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"}
Sep 11 21:56:25.372120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.371984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw" event={"ID":"440ab0fd-d73f-480d-8702-5d7a90778ca4","Type":"ContainerDied","Data":"4b8981fa0be83c0a1765e46f95c5ea4a741f5565e0791f8ad744008f14ea098b"}
Sep 11 21:56:25.372120 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.372008    2582 scope.go:117] "RemoveContainer" containerID="91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"
Sep 11 21:56:25.390801 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.390779    2582 scope.go:117] "RemoveContainer" containerID="071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"
Sep 11 21:56:25.399874 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.399845    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:56:25.400452 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.400435    2582 scope.go:117] "RemoveContainer" containerID="91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"
Sep 11 21:56:25.400805 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:25.400782    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498\": container with ID starting with 91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498 not found: ID does not exist" containerID="91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"
Sep 11 21:56:25.400892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.400814    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498"} err="failed to get container status \"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498\": rpc error: code = NotFound desc = could not find container \"91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498\": container with ID starting with 91022fc28e5e3c906367a8afffe69692979f61bf4a371d7c9708131b7c134498 not found: ID does not exist"
Sep 11 21:56:25.400892 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.400833    2582 scope.go:117] "RemoveContainer" containerID="071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"
Sep 11 21:56:25.401135 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:25.401115    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f\": container with ID starting with 071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f not found: ID does not exist" containerID="071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"
Sep 11 21:56:25.401173 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.401143    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f"} err="failed to get container status \"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f\": rpc error: code = NotFound desc = could not find container \"071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f\": container with ID starting with 071c8b4b610902d3b8d853d1b26c262d423eedc865179c0e2f973f0a738dc10f not found: ID does not exist"
Sep 11 21:56:25.405448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:25.405426    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1132e5-openshift-default-556cdcdfb6-tdrvw"]
Sep 11 21:56:26.455732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:26.455687    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" path="/var/lib/kubelet/pods/440ab0fd-d73f-480d-8702-5d7a90778ca4/volumes"
Sep 11 21:56:26.512932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:26.512888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:27.627658 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:27.627627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:27.965548 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:27.965447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:28.799489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:28.799457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:29.624650 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:29.624609    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:30.281639 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:30.281608    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:31.090164 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:31.090132    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:31.739792 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:31.739762    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:32.386750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:32.386687    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:32.700481 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:32.700398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:33.103489 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:33.103458    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:35.841846 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.841811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:35.866987 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.866955    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:56:35.867251 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.867223    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" podUID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" containerName="payload-processing" containerID="cri-o://7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687" gracePeriod=30
Sep 11 21:56:35.872051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.872022    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:56:35.872445 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.872416    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" podUID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" containerName="payload-pre-processing" containerID="cri-o://642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c" gracePeriod=30
Sep 11 21:56:35.906428 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.906397    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:35.975893 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:35.975858    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:36.041634 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.041590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:36.113056 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.112885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:36.173823 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.173796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:56:36.177028 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.177008    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:56:36.226626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.225656    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume\") pod \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") "
Sep 11 21:56:36.226626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.225769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc5gh\" (UniqueName: \"kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh\") pod \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\" (UID: \"672f2dd8-7b11-4020-9209-0fbaf891b2ba\") "
Sep 11 21:56:36.226626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.225812    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54hl5\" (UniqueName: \"kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5\") pod \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") "
Sep 11 21:56:36.226626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.226223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume\") pod \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\" (UID: \"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c\") "
Sep 11 21:56:36.226626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.226556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" (UID: "7f0ec857-4b5e-4c93-99ee-8c7001c7b97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:36.227080 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.226783    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:36.227296 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.227270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "672f2dd8-7b11-4020-9209-0fbaf891b2ba" (UID: "672f2dd8-7b11-4020-9209-0fbaf891b2ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:36.229071 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.229033    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5" (OuterVolumeSpecName: "kube-api-access-54hl5") pod "7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" (UID: "7f0ec857-4b5e-4c93-99ee-8c7001c7b97c"). InnerVolumeSpecName "kube-api-access-54hl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:36.229923 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.229895    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh" (OuterVolumeSpecName: "kube-api-access-fc5gh") pod "672f2dd8-7b11-4020-9209-0fbaf891b2ba" (UID: "672f2dd8-7b11-4020-9209-0fbaf891b2ba"). InnerVolumeSpecName "kube-api-access-fc5gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:36.328331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.328290    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc5gh\" (UniqueName: \"kubernetes.io/projected/672f2dd8-7b11-4020-9209-0fbaf891b2ba-kube-api-access-fc5gh\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:36.328331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.328323    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54hl5\" (UniqueName: \"kubernetes.io/projected/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c-kube-api-access-54hl5\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:36.328331 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.328332    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/672f2dd8-7b11-4020-9209-0fbaf891b2ba-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:36.373010 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.372916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:36.434947 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.434911    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" containerID="7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687" exitCode=0
Sep 11 21:56:36.435137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.434962    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"
Sep 11 21:56:36.435137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.434997    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" event={"ID":"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c","Type":"ContainerDied","Data":"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"}
Sep 11 21:56:36.435137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.435077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48" event={"ID":"7f0ec857-4b5e-4c93-99ee-8c7001c7b97c","Type":"ContainerDied","Data":"94eb1243066ecb008d5c3e2944e7aff57c8c986aa3bd1f5650fedbe115aee44c"}
Sep 11 21:56:36.435137 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.435110    2582 scope.go:117] "RemoveContainer" containerID="7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"
Sep 11 21:56:36.436406 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.436382    2582 generic.go:356] "Generic (PLEG): container finished" podID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" containerID="642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c" exitCode=0
Sep 11 21:56:36.436516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.436422    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"
Sep 11 21:56:36.436516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.436441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" event={"ID":"672f2dd8-7b11-4020-9209-0fbaf891b2ba","Type":"ContainerDied","Data":"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"}
Sep 11 21:56:36.436516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.436494    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq" event={"ID":"672f2dd8-7b11-4020-9209-0fbaf891b2ba","Type":"ContainerDied","Data":"a304eb369f7d1a7a6481713d4e67fe738d0f7f24b626171fb154ff9539e724fc"}
Sep 11 21:56:36.444681 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.444668    2582 scope.go:117] "RemoveContainer" containerID="7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"
Sep 11 21:56:36.444977 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:36.444958    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687\": container with ID starting with 7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687 not found: ID does not exist" containerID="7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"
Sep 11 21:56:36.445065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.444983    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687"} err="failed to get container status \"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687\": rpc error: code = NotFound desc = could not find container \"7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687\": container with ID starting with 7fa9d2de379edda79020f36891e7f1dbd7d398e64ca653f2bda50ffe75b0c687 not found: ID does not exist"
Sep 11 21:56:36.445065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.444999    2582 scope.go:117] "RemoveContainer" containerID="642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"
Sep 11 21:56:36.453570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.453555    2582 scope.go:117] "RemoveContainer" containerID="642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"
Sep 11 21:56:36.453866 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:36.453846    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c\": container with ID starting with 642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c not found: ID does not exist" containerID="642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"
Sep 11 21:56:36.453951 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.453870    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c"} err="failed to get container status \"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c\": rpc error: code = NotFound desc = could not find container \"642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c\": container with ID starting with 642f90bf7abb7b21c6045e0d5cdaeb758c22c3fa6f807e26ff5daa54df3b7d8c not found: ID does not exist"
Sep 11 21:56:36.460076 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.460054    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:56:36.464111 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.464085    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-24efea-7d5d9d647-fls48"]
Sep 11 21:56:36.473401 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.473374    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:56:36.480584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.480562    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-24efea-7d6c8bb4fd-wblxq"]
Sep 11 21:56:36.504113 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:36.504083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:37.188758 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:37.188616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:37.996428 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:37.996392    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:38.455533 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:38.455456    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" path="/var/lib/kubelet/pods/672f2dd8-7b11-4020-9209-0fbaf891b2ba/volumes"
Sep 11 21:56:38.455926 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:38.455824    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" path="/var/lib/kubelet/pods/7f0ec857-4b5e-4c93-99ee-8c7001c7b97c/volumes"
Sep 11 21:56:38.647971 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:38.647938    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:39.580558 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:39.580528    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:40.240494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:40.240448    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:42.342583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:42.342551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:42.405572 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:42.405517    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:42.609272 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:42.609179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.214614 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:43.214577    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:43.979583 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:43.979551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.629129 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:44.629095    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:44.864772 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:44.864740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:56:45.274629 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.274596    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.297779 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.297747    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:56:45.298073 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.298046    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" podUID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" containerName="payload-processing" containerID="cri-o://3967c8b43df84d85501e32406f595b773582dbd053038f13b63c30dc6b2af45e" gracePeriod=30
Sep 11 21:56:45.312310 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.312270    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:56:45.312562 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.312524    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" podUID="a9be77bd-b319-4984-a54d-f1021178f2ef" containerName="payload-pre-processing" containerID="cri-o://6a76205b778fb697517097dd91527b8ceb860edc90b718afbb6440a7913a8483" gracePeriod=30
Sep 11 21:56:45.343755 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.343633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.409854 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.409820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.491695 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.491653    2582 generic.go:356] "Generic (PLEG): container finished" podID="a9be77bd-b319-4984-a54d-f1021178f2ef" containerID="6a76205b778fb697517097dd91527b8ceb860edc90b718afbb6440a7913a8483" exitCode=0
Sep 11 21:56:45.491906 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.491807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" event={"ID":"a9be77bd-b319-4984-a54d-f1021178f2ef","Type":"ContainerDied","Data":"6a76205b778fb697517097dd91527b8ceb860edc90b718afbb6440a7913a8483"}
Sep 11 21:56:45.493938 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.493912    2582 generic.go:356] "Generic (PLEG): container finished" podID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" containerID="3967c8b43df84d85501e32406f595b773582dbd053038f13b63c30dc6b2af45e" exitCode=0
Sep 11 21:56:45.494055 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.493945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" event={"ID":"90cf08aa-6bc2-4ac8-ad14-d5e449adc697","Type":"ContainerDied","Data":"3967c8b43df84d85501e32406f595b773582dbd053038f13b63c30dc6b2af45e"}
Sep 11 21:56:45.518132 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.518100    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.612425 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.612394    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.612789 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.612766    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:56:45.620024 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.619998    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:56:45.713648 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.713597    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:45.725593 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.725550    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume\") pod \"a9be77bd-b319-4984-a54d-f1021178f2ef\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") "
Sep 11 21:56:45.725810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.725603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvl6m\" (UniqueName: \"kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m\") pod \"a9be77bd-b319-4984-a54d-f1021178f2ef\" (UID: \"a9be77bd-b319-4984-a54d-f1021178f2ef\") "
Sep 11 21:56:45.725810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.725702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume\") pod \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") "
Sep 11 21:56:45.725810 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.725762    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8xvz\" (UniqueName: \"kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz\") pod \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\" (UID: \"90cf08aa-6bc2-4ac8-ad14-d5e449adc697\") "
Sep 11 21:56:45.726051 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.726027    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9be77bd-b319-4984-a54d-f1021178f2ef" (UID: "a9be77bd-b319-4984-a54d-f1021178f2ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:45.726156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.726128    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9be77bd-b319-4984-a54d-f1021178f2ef-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:45.726156 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.726138    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90cf08aa-6bc2-4ac8-ad14-d5e449adc697" (UID: "90cf08aa-6bc2-4ac8-ad14-d5e449adc697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:45.728096 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.728070    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m" (OuterVolumeSpecName: "kube-api-access-mvl6m") pod "a9be77bd-b319-4984-a54d-f1021178f2ef" (UID: "a9be77bd-b319-4984-a54d-f1021178f2ef"). InnerVolumeSpecName "kube-api-access-mvl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:45.728344 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.728327    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz" (OuterVolumeSpecName: "kube-api-access-b8xvz") pod "90cf08aa-6bc2-4ac8-ad14-d5e449adc697" (UID: "90cf08aa-6bc2-4ac8-ad14-d5e449adc697"). InnerVolumeSpecName "kube-api-access-b8xvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:45.827395 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.827354    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvl6m\" (UniqueName: \"kubernetes.io/projected/a9be77bd-b319-4984-a54d-f1021178f2ef-kube-api-access-mvl6m\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:45.827395 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.827402    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:45.827613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.827418    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xvz\" (UniqueName: \"kubernetes.io/projected/90cf08aa-6bc2-4ac8-ad14-d5e449adc697-kube-api-access-b8xvz\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:45.843150 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:45.843044    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:46.421767 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.421731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:46.500136 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.500108    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"
Sep 11 21:56:46.500343 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.500139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk" event={"ID":"a9be77bd-b319-4984-a54d-f1021178f2ef","Type":"ContainerDied","Data":"6df9fbaff79706cbf9a68cf4cb0e726835ed826e67c7318b2a643edf1b35a875"}
Sep 11 21:56:46.500343 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.500236    2582 scope.go:117] "RemoveContainer" containerID="6a76205b778fb697517097dd91527b8ceb860edc90b718afbb6440a7913a8483"
Sep 11 21:56:46.501780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.501493    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"
Sep 11 21:56:46.501780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.501550    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v" event={"ID":"90cf08aa-6bc2-4ac8-ad14-d5e449adc697","Type":"ContainerDied","Data":"0a27833f08d3676b19168ab298f86ee632c6781a475f7da55c4fa2ec51079e56"}
Sep 11 21:56:46.509609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.509580    2582 scope.go:117] "RemoveContainer" containerID="3967c8b43df84d85501e32406f595b773582dbd053038f13b63c30dc6b2af45e"
Sep 11 21:56:46.518609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.518588    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:56:46.525864 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.525837    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f12741-7c879744b6-fksdk"]
Sep 11 21:56:46.534831 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.534799    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:56:46.538844 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:46.538799    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f12741-7ff4fd4fbf-fv66v"]
Sep 11 21:56:47.180918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:47.180873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:48.209291 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:48.209251    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:48.455841 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:48.455808    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" path="/var/lib/kubelet/pods/90cf08aa-6bc2-4ac8-ad14-d5e449adc697/volumes"
Sep 11 21:56:48.456159 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:48.456146    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9be77bd-b319-4984-a54d-f1021178f2ef" path="/var/lib/kubelet/pods/a9be77bd-b319-4984-a54d-f1021178f2ef/volumes"
Sep 11 21:56:48.913523 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:48.913490    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:49.775792 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:49.775692    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:50.420012 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:50.419972    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:50.674466 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:50.674359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:51.275957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:51.275917    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:51.938694 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:51.938650    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:52.012185 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:52.012152    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:52.098582 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:52.098534    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:52.217920 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:52.217804    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:52.861540 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:52.861499    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:53.582829 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:53.582791    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.129305 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.129266    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:56:54.129777 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.129555    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" podUID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" containerName="maas-api" containerID="cri-o://d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1" gracePeriod=30
Sep 11 21:56:54.153651 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.153620    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.176607 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.176572    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:56:54.176902 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.176839    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" podUID="1a987da3-932a-4dc6-931a-72089eae4097" containerName="payload-processing" containerID="cri-o://24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691" gracePeriod=30
Sep 11 21:56:54.184838 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.184813    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:56:54.185070 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.185042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" podUID="d834416f-4984-4e12-bb78-7aad45363ad1" containerName="payload-pre-processing" containerID="cri-o://d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad" gracePeriod=30
Sep 11 21:56:54.219566 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.219533    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.287059 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.287020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.360065 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.360020    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.421866 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.421820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.479310 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.479277    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:56:54.524449 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.524412    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:56:54.527584 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.527560    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:56:54.547546 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.547518    2582 generic.go:356] "Generic (PLEG): container finished" podID="d834416f-4984-4e12-bb78-7aad45363ad1" containerID="d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad" exitCode=0
Sep 11 21:56:54.547734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.547556    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"
Sep 11 21:56:54.547734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.547613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" event={"ID":"d834416f-4984-4e12-bb78-7aad45363ad1","Type":"ContainerDied","Data":"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"}
Sep 11 21:56:54.547734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.547675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444" event={"ID":"d834416f-4984-4e12-bb78-7aad45363ad1","Type":"ContainerDied","Data":"5e39eedc3b860b81cdce14bf4bc0090e48c165cdf62e6e58e8154be2d482c52a"}
Sep 11 21:56:54.547734 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.547702    2582 scope.go:117] "RemoveContainer" containerID="d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"
Sep 11 21:56:54.548921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.548901    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a987da3-932a-4dc6-931a-72089eae4097" containerID="24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691" exitCode=0
Sep 11 21:56:54.549042 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.548993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" event={"ID":"1a987da3-932a-4dc6-931a-72089eae4097","Type":"ContainerDied","Data":"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"}
Sep 11 21:56:54.549042 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.549025    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"
Sep 11 21:56:54.549217 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.549043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr" event={"ID":"1a987da3-932a-4dc6-931a-72089eae4097","Type":"ContainerDied","Data":"6ab4e4c8136ad27860e2ca14625fa38d375c4e3f0a5c0a60c7fdd3dad5739f9c"}
Sep 11 21:56:54.552286 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.552240    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" containerID="d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1" exitCode=0
Sep 11 21:56:54.552403 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.552288    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"
Sep 11 21:56:54.552518 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.552481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" event={"ID":"dc405f57-f3e2-4707-aedf-f97d09b6c3f1","Type":"ContainerDied","Data":"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"}
Sep 11 21:56:54.552638 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.552545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx" event={"ID":"dc405f57-f3e2-4707-aedf-f97d09b6c3f1","Type":"ContainerDied","Data":"eeb2c92d4408126a5db1c28470e66e0794cfec679545ab242b0a1f4886438803"}
Sep 11 21:56:54.559118 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.559094    2582 scope.go:117] "RemoveContainer" containerID="d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"
Sep 11 21:56:54.559480 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:54.559453    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad\": container with ID starting with d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad not found: ID does not exist" containerID="d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"
Sep 11 21:56:54.559570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.559491    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad"} err="failed to get container status \"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad\": rpc error: code = NotFound desc = could not find container \"d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad\": container with ID starting with d184cac3f99a8c7f261aa96f240da62aa4ad53a40cbee0e21699f79fa38fd0ad not found: ID does not exist"
Sep 11 21:56:54.559570 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.559515    2582 scope.go:117] "RemoveContainer" containerID="24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"
Sep 11 21:56:54.569621 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.569601    2582 scope.go:117] "RemoveContainer" containerID="24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"
Sep 11 21:56:54.569948 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:54.569932    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691\": container with ID starting with 24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691 not found: ID does not exist" containerID="24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"
Sep 11 21:56:54.570001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.569952    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691"} err="failed to get container status \"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691\": rpc error: code = NotFound desc = could not find container \"24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691\": container with ID starting with 24e0f38edcc0ffc44b58cab8985f98f759ceca90a8ee40b3ad9654147ba2c691 not found: ID does not exist"
Sep 11 21:56:54.570001 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.569974    2582 scope.go:117] "RemoveContainer" containerID="d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"
Sep 11 21:56:54.579074 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.579055    2582 scope.go:117] "RemoveContainer" containerID="d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"
Sep 11 21:56:54.579358 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:56:54.579343    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1\": container with ID starting with d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1 not found: ID does not exist" containerID="d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"
Sep 11 21:56:54.579420 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.579364    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1"} err="failed to get container status \"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1\": rpc error: code = NotFound desc = could not find container \"d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1\": container with ID starting with d7e1dcb69415e197df7050f68c61f013338f47bb12e1e8c37b155dc3bbd04ff1 not found: ID does not exist"
Sep 11 21:56:54.616863 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.616831    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs\") pod \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.616874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cn4x\" (UniqueName: \"kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x\") pod \"1a987da3-932a-4dc6-931a-72089eae4097\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.616901    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") pod \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.616918    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7kml\" (UniqueName: \"kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml\") pod \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.616976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca\") pod \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\" (UID: \"dc405f57-f3e2-4707-aedf-f97d09b6c3f1\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617016    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume\") pod \"d834416f-4984-4e12-bb78-7aad45363ad1\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") "
Sep 11 21:56:54.617039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617038    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j84f8\" (UniqueName: \"kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8\") pod \"d834416f-4984-4e12-bb78-7aad45363ad1\" (UID: \"d834416f-4984-4e12-bb78-7aad45363ad1\") "
Sep 11 21:56:54.617448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617075    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume\") pod \"1a987da3-932a-4dc6-931a-72089eae4097\" (UID: \"1a987da3-932a-4dc6-931a-72089eae4097\") "
Sep 11 21:56:54.617448 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617367    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc405f57-f3e2-4707-aedf-f97d09b6c3f1" (UID: "dc405f57-f3e2-4707-aedf-f97d09b6c3f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:54.617565 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617480    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.617805 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.617627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d834416f-4984-4e12-bb78-7aad45363ad1" (UID: "d834416f-4984-4e12-bb78-7aad45363ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:54.618128 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.618100    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a987da3-932a-4dc6-931a-72089eae4097" (UID: "1a987da3-932a-4dc6-931a-72089eae4097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:56:54.619441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.619420    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml" (OuterVolumeSpecName: "kube-api-access-p7kml") pod "dc405f57-f3e2-4707-aedf-f97d09b6c3f1" (UID: "dc405f57-f3e2-4707-aedf-f97d09b6c3f1"). InnerVolumeSpecName "kube-api-access-p7kml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:54.619441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.619428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc405f57-f3e2-4707-aedf-f97d09b6c3f1" (UID: "dc405f57-f3e2-4707-aedf-f97d09b6c3f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:54.619441 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.619428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8" (OuterVolumeSpecName: "kube-api-access-j84f8") pod "d834416f-4984-4e12-bb78-7aad45363ad1" (UID: "d834416f-4984-4e12-bb78-7aad45363ad1"). InnerVolumeSpecName "kube-api-access-j84f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:54.619862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.619845    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc405f57-f3e2-4707-aedf-f97d09b6c3f1" (UID: "dc405f57-f3e2-4707-aedf-f97d09b6c3f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:56:54.619862 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.619851    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x" (OuterVolumeSpecName: "kube-api-access-9cn4x") pod "1a987da3-932a-4dc6-931a-72089eae4097" (UID: "1a987da3-932a-4dc6-931a-72089eae4097"). InnerVolumeSpecName "kube-api-access-9cn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718160    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d834416f-4984-4e12-bb78-7aad45363ad1-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718193    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j84f8\" (UniqueName: \"kubernetes.io/projected/d834416f-4984-4e12-bb78-7aad45363ad1-kube-api-access-j84f8\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718203    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a987da3-932a-4dc6-931a-72089eae4097-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718214    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718223    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cn4x\" (UniqueName: \"kubernetes.io/projected/1a987da3-932a-4dc6-931a-72089eae4097-kube-api-access-9cn4x\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718233    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.718256 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.718241    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kml\" (UniqueName: \"kubernetes.io/projected/dc405f57-f3e2-4707-aedf-f97d09b6c3f1-kube-api-access-p7kml\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:56:54.734329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.734295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:54.874957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.874927    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:56:54.878867 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.878841    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5cb5f4-7bb6676fcd-j7444"]
Sep 11 21:56:54.891507 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.891479    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:56:54.893494 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.893467    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5cb5f4-5d689897f-pshrr"]
Sep 11 21:56:54.902604 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.902577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:56:54.905945 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:54.905918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5cb5f4-7c489c696c-5vbcx"]
Sep 11 21:56:55.219903 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:55.219872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:55.748349 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:55.748301    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:55.951453 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:55.951424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:56.454471 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:56.454437    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a987da3-932a-4dc6-931a-72089eae4097" path="/var/lib/kubelet/pods/1a987da3-932a-4dc6-931a-72089eae4097/volumes"
Sep 11 21:56:56.454850 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:56.454804    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d834416f-4984-4e12-bb78-7aad45363ad1" path="/var/lib/kubelet/pods/d834416f-4984-4e12-bb78-7aad45363ad1/volumes"
Sep 11 21:56:56.455163 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:56.455150    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" path="/var/lib/kubelet/pods/dc405f57-f3e2-4707-aedf-f97d09b6c3f1/volumes"
Sep 11 21:56:56.688412 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:56.688360    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:57.385397 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:57.385368    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:56:58.203564 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:56:58.203524    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:00.600010 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:00.599974    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:00.665618 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:00.665586    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:00.881357 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:00.881266    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:00.950949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:00.950912    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:01.205315 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205229    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"]
Sep 11 21:57:01.205649 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205636    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d834416f-4984-4e12-bb78-7aad45363ad1" containerName="payload-pre-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205650    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d834416f-4984-4e12-bb78-7aad45363ad1" containerName="payload-pre-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205661    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9be77bd-b319-4984-a54d-f1021178f2ef" containerName="payload-pre-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205666    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9be77bd-b319-4984-a54d-f1021178f2ef" containerName="payload-pre-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205673    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" containerName="payload-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205680    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" containerName="payload-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205687    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" containerName="payload-pre-processing"
Sep 11 21:57:01.205693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205692    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205702    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a987da3-932a-4dc6-931a-72089eae4097" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205709    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a987da3-932a-4dc6-931a-72089eae4097" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205733    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205738    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205748    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205753    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205759    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea45ee7-a958-4b79-8599-19bbc3e02889" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205764    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea45ee7-a958-4b79-8599-19bbc3e02889" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205770    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205775    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205781    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205785    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205796    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205801    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205858    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a987da3-932a-4dc6-931a-72089eae4097" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205865    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d834416f-4984-4e12-bb78-7aad45363ad1" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205871    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9be77bd-b319-4984-a54d-f1021178f2ef" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205881    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="672f2dd8-7b11-4020-9209-0fbaf891b2ba" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205888    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc405f57-f3e2-4707-aedf-f97d09b6c3f1" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205892    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea45ee7-a958-4b79-8599-19bbc3e02889" containerName="maas-api"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205897    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cf08aa-6bc2-4ac8-ad14-d5e449adc697" containerName="payload-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205902    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2520c489-1e77-4bb3-8661-d3b3c8cad2c8" containerName="payload-pre-processing"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205906    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205912    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="440ab0fd-d73f-480d-8702-5d7a90778ca4" containerName="istio-proxy"
Sep 11 21:57:01.205957 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.205917    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0ec857-4b5e-4c93-99ee-8c7001c7b97c" containerName="payload-processing"
Sep 11 21:57:01.210824 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.210804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:01.213061 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.212989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 21:57:01.213061 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.212994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 21:57:01.216732 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.216682    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"]
Sep 11 21:57:01.386181 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.386139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm\" (UID: \"44b02bdc-b7a9-4299-8145-5cda4d27571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:01.487703 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.487664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm\" (UID: \"44b02bdc-b7a9-4299-8145-5cda4d27571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:01.488326 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.488305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm\" (UID: \"44b02bdc-b7a9-4299-8145-5cda4d27571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:01.524198 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.524160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:01.702147 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.702113    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:01.799562 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:01.799537    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"]
Sep 11 21:57:01.802052 ip-10-0-141-125 kubenswrapper[2582]: W0911 21:57:01.802024    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b02bdc_b7a9_4299_8145_5cda4d27571b.slice/crio-08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155 WatchSource:0}: Error finding container 08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155: Status 404 returned error can't find the container with id 08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155
Sep 11 21:57:02.493963 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:02.493926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:02.602921 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:02.602883    2582 generic.go:356] "Generic (PLEG): container finished" podID="44b02bdc-b7a9-4299-8145-5cda4d27571b" containerID="4335475233faf0a1439ba29ed34955393c4c9e012597099508fa21ff2dbd4a10" exitCode=0
Sep 11 21:57:02.603094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:02.602928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm" event={"ID":"44b02bdc-b7a9-4299-8145-5cda4d27571b","Type":"ContainerDied","Data":"4335475233faf0a1439ba29ed34955393c4c9e012597099508fa21ff2dbd4a10"}
Sep 11 21:57:02.603094 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:02.602981    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm" event={"ID":"44b02bdc-b7a9-4299-8145-5cda4d27571b","Type":"ContainerStarted","Data":"08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155"}
Sep 11 21:57:03.220686 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:03.220644    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:03.741119 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:03.741090    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:03.888383 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:03.888294    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:03.911362 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:03.911327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca\") pod \"44b02bdc-b7a9-4299-8145-5cda4d27571b\" (UID: \"44b02bdc-b7a9-4299-8145-5cda4d27571b\") "
Sep 11 21:57:03.911692 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:03.911669    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44b02bdc-b7a9-4299-8145-5cda4d27571b" (UID: "44b02bdc-b7a9-4299-8145-5cda4d27571b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:04.012753 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:04.012696    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44b02bdc-b7a9-4299-8145-5cda4d27571b-maas-api-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:04.613675 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:04.613635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm" event={"ID":"44b02bdc-b7a9-4299-8145-5cda4d27571b","Type":"ContainerDied","Data":"08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155"}
Sep 11 21:57:04.613675 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:04.613666    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"
Sep 11 21:57:04.614121 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:04.613695    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08135cf7ec0378c31a7a79c3280de7a04f20b444b97bf4e474c73b9caa508155"
Sep 11 21:57:09.343880 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.343843    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.403709 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.403666    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.538946 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.538907    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.574780 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.574747    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:57:09.575127 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.575017    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" podUID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" containerName="payload-processing" containerID="cri-o://b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f" gracePeriod=30
Sep 11 21:57:09.582260 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.582226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:57:09.582526 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.582490    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" podUID="8b104980-a2df-41d5-8311-0156528795e5" containerName="payload-pre-processing" containerID="cri-o://6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed" gracePeriod=30
Sep 11 21:57:09.624034 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.624000    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.693385 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.693351    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.787932 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.787896    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.850008 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.849914    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:09.854031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.854007    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:57:09.859106 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.859076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume\") pod \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") "
Sep 11 21:57:09.859250 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.859153    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85wbh\" (UniqueName: \"kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh\") pod \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\" (UID: \"99ccb7a2-5993-4285-ba13-ce281c34e6dd\") "
Sep 11 21:57:09.859485 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.859459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99ccb7a2-5993-4285-ba13-ce281c34e6dd" (UID: "99ccb7a2-5993-4285-ba13-ce281c34e6dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:09.861426 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.861406    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh" (OuterVolumeSpecName: "kube-api-access-85wbh") pod "99ccb7a2-5993-4285-ba13-ce281c34e6dd" (UID: "99ccb7a2-5993-4285-ba13-ce281c34e6dd"). InnerVolumeSpecName "kube-api-access-85wbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:09.877889 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.877860    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:57:09.959685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.959645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n2jv\" (UniqueName: \"kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv\") pod \"8b104980-a2df-41d5-8311-0156528795e5\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") "
Sep 11 21:57:09.959685 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.959684    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume\") pod \"8b104980-a2df-41d5-8311-0156528795e5\" (UID: \"8b104980-a2df-41d5-8311-0156528795e5\") "
Sep 11 21:57:09.959949 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.959937    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99ccb7a2-5993-4285-ba13-ce281c34e6dd-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:09.959986 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.959951    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wbh\" (UniqueName: \"kubernetes.io/projected/99ccb7a2-5993-4285-ba13-ce281c34e6dd-kube-api-access-85wbh\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:09.960091 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.960069    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b104980-a2df-41d5-8311-0156528795e5" (UID: "8b104980-a2df-41d5-8311-0156528795e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:09.961935 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:09.961916    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv" (OuterVolumeSpecName: "kube-api-access-4n2jv") pod "8b104980-a2df-41d5-8311-0156528795e5" (UID: "8b104980-a2df-41d5-8311-0156528795e5"). InnerVolumeSpecName "kube-api-access-4n2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:10.061192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.061154    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n2jv\" (UniqueName: \"kubernetes.io/projected/8b104980-a2df-41d5-8311-0156528795e5-kube-api-access-4n2jv\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:10.061192 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.061195    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b104980-a2df-41d5-8311-0156528795e5-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:10.126571 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.126465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:10.267048 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.267012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:10.648276 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.648241    2582 generic.go:356] "Generic (PLEG): container finished" podID="8b104980-a2df-41d5-8311-0156528795e5" containerID="6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed" exitCode=0
Sep 11 21:57:10.648750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.648289    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"
Sep 11 21:57:10.648750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.648326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" event={"ID":"8b104980-a2df-41d5-8311-0156528795e5","Type":"ContainerDied","Data":"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"}
Sep 11 21:57:10.648750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.648382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv" event={"ID":"8b104980-a2df-41d5-8311-0156528795e5","Type":"ContainerDied","Data":"4ebda89366d3d102b9c8a7ea1b339cd07cf3cfc7493575839dcedb547ab92ac1"}
Sep 11 21:57:10.648750 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.648412    2582 scope.go:117] "RemoveContainer" containerID="6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"
Sep 11 21:57:10.649875 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.649831    2582 generic.go:356] "Generic (PLEG): container finished" podID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" containerID="b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f" exitCode=0
Sep 11 21:57:10.649945 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.649881    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"
Sep 11 21:57:10.650003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.649931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" event={"ID":"99ccb7a2-5993-4285-ba13-ce281c34e6dd","Type":"ContainerDied","Data":"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"}
Sep 11 21:57:10.650003 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.649987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4" event={"ID":"99ccb7a2-5993-4285-ba13-ce281c34e6dd","Type":"ContainerDied","Data":"c3ea9f37d125bbb3aac8432ffdd2a755e892697097bed8a63de8352fbe9bd076"}
Sep 11 21:57:10.657855 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.657839    2582 scope.go:117] "RemoveContainer" containerID="6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"
Sep 11 21:57:10.658153 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:57:10.658130    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed\": container with ID starting with 6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed not found: ID does not exist" containerID="6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"
Sep 11 21:57:10.658199 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.658166    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed"} err="failed to get container status \"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed\": rpc error: code = NotFound desc = could not find container \"6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed\": container with ID starting with 6fd447d28ec7161f7c32f6c523b7531d9595405f9dc5595e1c6a5f47abb93bed not found: ID does not exist"
Sep 11 21:57:10.658199 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.658190    2582 scope.go:117] "RemoveContainer" containerID="b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"
Sep 11 21:57:10.666622 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.666607    2582 scope.go:117] "RemoveContainer" containerID="b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"
Sep 11 21:57:10.666940 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:57:10.666918    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f\": container with ID starting with b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f not found: ID does not exist" containerID="b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"
Sep 11 21:57:10.667031 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.666944    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f"} err="failed to get container status \"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f\": rpc error: code = NotFound desc = could not find container \"b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f\": container with ID starting with b2d70f4a6ba849bdfcae001021f9ade9a4b9c7ebe9935a0eabf95c118d4c1c1f not found: ID does not exist"
Sep 11 21:57:10.667693 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.667676    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:57:10.674011 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.673973    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-35e777-7fcbcbfcbc-x2mq4"]
Sep 11 21:57:10.683248 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.683230    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:57:10.687391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.687371    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-35e777-85bfc845c8-7m9tv"]
Sep 11 21:57:10.983477 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:10.983441    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:11.073073 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:11.073036    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:11.809267 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:11.809229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.456231 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.456199    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b104980-a2df-41d5-8311-0156528795e5" path="/var/lib/kubelet/pods/8b104980-a2df-41d5-8311-0156528795e5/volumes"
Sep 11 21:57:12.456803 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.456780    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" path="/var/lib/kubelet/pods/99ccb7a2-5993-4285-ba13-ce281c34e6dd/volumes"
Sep 11 21:57:12.467229 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.467197    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.494277 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.494243    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:57:12.494534 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.494506    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" podUID="b6946fab-2dd3-4f86-877d-247165a21c19" containerName="payload-processing" containerID="cri-o://c23d62166a01594d884467662f84f92ff908e105f1c274842617974fca8d9482" gracePeriod=30
Sep 11 21:57:12.514928 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.514899    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:57:12.515172 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.515143    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" podUID="965cdd75-56af-409e-a4dc-8e6060dae9a6" containerName="payload-pre-processing" containerID="cri-o://82eae41930a79bfeb261b1823d67695fae187c9843ccb1f77c639c1d8edd19be" gracePeriod=30
Sep 11 21:57:12.536623 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.536590    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.604390 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.604357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.659678 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.659644    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"]
Sep 11 21:57:12.662900 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.662870    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-afd794-z57tm"]
Sep 11 21:57:12.666939 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.666899    2582 generic.go:356] "Generic (PLEG): container finished" podID="b6946fab-2dd3-4f86-877d-247165a21c19" containerID="c23d62166a01594d884467662f84f92ff908e105f1c274842617974fca8d9482" exitCode=0
Sep 11 21:57:12.667075 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.666955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" event={"ID":"b6946fab-2dd3-4f86-877d-247165a21c19","Type":"ContainerDied","Data":"c23d62166a01594d884467662f84f92ff908e105f1c274842617974fca8d9482"}
Sep 11 21:57:12.668970 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.668949    2582 generic.go:356] "Generic (PLEG): container finished" podID="965cdd75-56af-409e-a4dc-8e6060dae9a6" containerID="82eae41930a79bfeb261b1823d67695fae187c9843ccb1f77c639c1d8edd19be" exitCode=0
Sep 11 21:57:12.669114 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.669015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" event={"ID":"965cdd75-56af-409e-a4dc-8e6060dae9a6","Type":"ContainerDied","Data":"82eae41930a79bfeb261b1823d67695fae187c9843ccb1f77c639c1d8edd19be"}
Sep 11 21:57:12.705251 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.705202    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:12.779197 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.779169    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:57:12.806649 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.806622    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:57:12.889056 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume\") pod \"965cdd75-56af-409e-a4dc-8e6060dae9a6\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") "
Sep 11 21:57:12.889056 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msljl\" (UniqueName: \"kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl\") pod \"965cdd75-56af-409e-a4dc-8e6060dae9a6\" (UID: \"965cdd75-56af-409e-a4dc-8e6060dae9a6\") "
Sep 11 21:57:12.889502 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889094    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume\") pod \"b6946fab-2dd3-4f86-877d-247165a21c19\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") "
Sep 11 21:57:12.889502 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889112    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rftqb\" (UniqueName: \"kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb\") pod \"b6946fab-2dd3-4f86-877d-247165a21c19\" (UID: \"b6946fab-2dd3-4f86-877d-247165a21c19\") "
Sep 11 21:57:12.889626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889518    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "965cdd75-56af-409e-a4dc-8e6060dae9a6" (UID: "965cdd75-56af-409e-a4dc-8e6060dae9a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:12.889626 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.889520    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6946fab-2dd3-4f86-877d-247165a21c19" (UID: "b6946fab-2dd3-4f86-877d-247165a21c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:12.891395 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.891371    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl" (OuterVolumeSpecName: "kube-api-access-msljl") pod "965cdd75-56af-409e-a4dc-8e6060dae9a6" (UID: "965cdd75-56af-409e-a4dc-8e6060dae9a6"). InnerVolumeSpecName "kube-api-access-msljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:12.891534 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.891415    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb" (OuterVolumeSpecName: "kube-api-access-rftqb") pod "b6946fab-2dd3-4f86-877d-247165a21c19" (UID: "b6946fab-2dd3-4f86-877d-247165a21c19"). InnerVolumeSpecName "kube-api-access-rftqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:12.990682 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.990646    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/965cdd75-56af-409e-a4dc-8e6060dae9a6-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:12.990682 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.990679    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msljl\" (UniqueName: \"kubernetes.io/projected/965cdd75-56af-409e-a4dc-8e6060dae9a6-kube-api-access-msljl\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:12.990907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.990692    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6946fab-2dd3-4f86-877d-247165a21c19-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:12.990907 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.990704    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rftqb\" (UniqueName: \"kubernetes.io/projected/b6946fab-2dd3-4f86-877d-247165a21c19-kube-api-access-rftqb\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:57:12.998956 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:12.998916    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:13.109498 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.109465    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:13.675775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.675740    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"
Sep 11 21:57:13.675980 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.675744    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4" event={"ID":"b6946fab-2dd3-4f86-877d-247165a21c19","Type":"ContainerDied","Data":"119a749a39bbe0823d7ee948558f64c941d44edf721ed40f53d8662a18cc4046"}
Sep 11 21:57:13.675980 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.675892    2582 scope.go:117] "RemoveContainer" containerID="c23d62166a01594d884467662f84f92ff908e105f1c274842617974fca8d9482"
Sep 11 21:57:13.677236 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.677107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf" event={"ID":"965cdd75-56af-409e-a4dc-8e6060dae9a6","Type":"ContainerDied","Data":"e10c2ccd0f2c4a01a857066dfdd7e187289a29fb2962ec70fba9142f7fb5dcee"}
Sep 11 21:57:13.677236 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.677149    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"
Sep 11 21:57:13.688048 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.687980    2582 scope.go:117] "RemoveContainer" containerID="82eae41930a79bfeb261b1823d67695fae187c9843ccb1f77c639c1d8edd19be"
Sep 11 21:57:13.701555 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.701532    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:57:13.705784 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.705764    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-afd794-6b8ff8b5cc-m7sq4"]
Sep 11 21:57:13.722918 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.722885    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:57:13.724529 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.724504    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-afd794-86859f975d-6zsqf"]
Sep 11 21:57:13.957689 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:13.957595    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:14.059077 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:14.059048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:14.455415 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:14.455374    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b02bdc-b7a9-4299-8145-5cda4d27571b" path="/var/lib/kubelet/pods/44b02bdc-b7a9-4299-8145-5cda4d27571b/volumes"
Sep 11 21:57:14.455775 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:14.455761    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965cdd75-56af-409e-a4dc-8e6060dae9a6" path="/var/lib/kubelet/pods/965cdd75-56af-409e-a4dc-8e6060dae9a6/volumes"
Sep 11 21:57:14.456102 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:14.456090    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6946fab-2dd3-4f86-877d-247165a21c19" path="/var/lib/kubelet/pods/b6946fab-2dd3-4f86-877d-247165a21c19/volumes"
Sep 11 21:57:14.656409 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:14.656318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.788760 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:15.788702    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:15.956347 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:15.956305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:16.762678 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:16.762636    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:16.835381 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:16.835312    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:17.058391 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:17.058291    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:17.951523 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:17.951479    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:18.727936 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:18.727899    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:18.791996 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:18.791963    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:18.994475 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:18.994382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:21.529844 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:21.529811    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:21.914100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:21.914002    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:22.958270 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:22.958237    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:57:26.892063 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:26.891969    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:27.153934 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:27.153838    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:28.012983 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:28.012948    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:36.255180 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:36.255146    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:57:44.472461 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:44.472423    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:44.572288 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:44.572256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:57:48.958307 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:57:48.958271    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:58:01.363836 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:01.363802    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:58:01.893943 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:01.893904    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:06.317609 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:06.317574    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:07.281613 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:07.281579    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:58:07.312312 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:07.312281    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:12.094961 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:12.094931    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:15.060324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:15.060286    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:58:22.303324 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.303263    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" podUID="5d11922b-fd01-423f-872c-2fd21b8018a2" containerName="main" containerID="cri-o://dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f" gracePeriod=45
Sep 11 21:58:22.554100 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.554041    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:58:22.572439 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572469    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572570    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572601 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572653    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd6r2\" (UniqueName: \"kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2\") pod \"5d11922b-fd01-423f-872c-2fd21b8018a2\" (UID: \"5d11922b-fd01-423f-872c-2fd21b8018a2\") "
Sep 11 21:58:22.572804 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home" (OuterVolumeSpecName: "home") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:22.572975 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.572951    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:22.573153 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.573067    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-tmp-dir\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.573153 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.573089    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-home\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.573305 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.573196    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:22.575688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.575663    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:58:22.575688 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.575673    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm" (OuterVolumeSpecName: "dshm") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:58:22.578560 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.578532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2" (OuterVolumeSpecName: "kube-api-access-kd6r2") pod "5d11922b-fd01-423f-872c-2fd21b8018a2" (UID: "5d11922b-fd01-423f-872c-2fd21b8018a2"). InnerVolumeSpecName "kube-api-access-kd6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:58:22.674419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.674382    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd6r2\" (UniqueName: \"kubernetes.io/projected/5d11922b-fd01-423f-872c-2fd21b8018a2-kube-api-access-kd6r2\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.674419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.674409    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d11922b-fd01-423f-872c-2fd21b8018a2-tls-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.674419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.674418    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-model-cache\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:22.674419 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:22.674426    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d11922b-fd01-423f-872c-2fd21b8018a2-dshm\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 21:58:23.025994 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.025960    2582 generic.go:356] "Generic (PLEG): container finished" podID="5d11922b-fd01-423f-872c-2fd21b8018a2" containerID="dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f" exitCode=0
Sep 11 21:58:23.026179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.026004    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"
Sep 11 21:58:23.026179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.026018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" event={"ID":"5d11922b-fd01-423f-872c-2fd21b8018a2","Type":"ContainerDied","Data":"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"}
Sep 11 21:58:23.026179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.026061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv" event={"ID":"5d11922b-fd01-423f-872c-2fd21b8018a2","Type":"ContainerDied","Data":"ca23d8860f9ebf5c55a087f415d09f0ad92f7e82a7c140be11046dfde84c0b92"}
Sep 11 21:58:23.026179 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.026083    2582 scope.go:117] "RemoveContainer" containerID="dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"
Sep 11 21:58:23.035039 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.035018    2582 scope.go:117] "RemoveContainer" containerID="dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"
Sep 11 21:58:23.035290 ip-10-0-141-125 kubenswrapper[2582]: E0911 21:58:23.035274    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f\": container with ID starting with dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f not found: ID does not exist" containerID="dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"
Sep 11 21:58:23.035345 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.035299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f"} err="failed to get container status \"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f\": rpc error: code = NotFound desc = could not find container \"dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f\": container with ID starting with dbaf426ecde21ef03aa5d6c4008e62bcde29f707838ef9584e571ea48c0ad77f not found: ID does not exist"
Sep 11 21:58:23.045891 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.045867    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:58:23.049257 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:23.049237    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4e0323/auth-test-model-4e0323-kserve-cc6cb8cc5-5tkhv"]
Sep 11 21:58:24.454410 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:24.454379    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d11922b-fd01-423f-872c-2fd21b8018a2" path="/var/lib/kubelet/pods/5d11922b-fd01-423f-872c-2fd21b8018a2/volumes"
Sep 11 21:58:30.676829 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:30.676795    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:58:34.078516 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:34.078471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:58:59.758707 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:58:59.758616    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:59:02.759329 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:02.759295    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:14.661997 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:14.661912    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:59:16.547675 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:16.547627    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:17.415000 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:17.414961    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:25.757220 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:25.757182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:59:34.577618 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:34.577582    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:38.071254 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:38.071220    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 21:59:41.953851 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:41.953816    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:42.054614 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:42.054569    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:57.709554 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:57.709512    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 21:59:57.810924 ip-10-0-141-125 kubenswrapper[2582]: I0911 21:59:57.810888    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:00.137197 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137165    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"]
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137528    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d11922b-fd01-423f-872c-2fd21b8018a2" containerName="main"
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137538    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d11922b-fd01-423f-872c-2fd21b8018a2" containerName="main"
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137548    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" containerName="payload-processing"
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137553    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" containerName="payload-processing"
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137560    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6946fab-2dd3-4f86-877d-247165a21c19" containerName="payload-processing"
Sep 11 22:00:00.137582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137565    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6946fab-2dd3-4f86-877d-247165a21c19" containerName="payload-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137593    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965cdd75-56af-409e-a4dc-8e6060dae9a6" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137600    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="965cdd75-56af-409e-a4dc-8e6060dae9a6" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137614    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b104980-a2df-41d5-8311-0156528795e5" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137620    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b104980-a2df-41d5-8311-0156528795e5" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137631    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b02bdc-b7a9-4299-8145-5cda4d27571b" containerName="revoke-keys"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137636    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b02bdc-b7a9-4299-8145-5cda4d27571b" containerName="revoke-keys"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137692    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="965cdd75-56af-409e-a4dc-8e6060dae9a6" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137701    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b104980-a2df-41d5-8311-0156528795e5" containerName="payload-pre-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137707    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="99ccb7a2-5993-4285-ba13-ce281c34e6dd" containerName="payload-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137738    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b02bdc-b7a9-4299-8145-5cda4d27571b" containerName="revoke-keys"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137745    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6946fab-2dd3-4f86-877d-247165a21c19" containerName="payload-processing"
Sep 11 22:00:00.137818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.137752    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d11922b-fd01-423f-872c-2fd21b8018a2" containerName="main"
Sep 11 22:00:00.140901 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.140884    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"]
Sep 11 22:00:00.141033 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.141012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"
Sep 11 22:00:00.143295 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.143271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:00:00.144287 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.144272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"
Sep 11 22:00:00.152193 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.152144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"]
Sep 11 22:00:00.152894 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.152872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"
Sep 11 22:00:00.154590 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.154567    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"]
Sep 11 22:00:00.162235 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.162214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"
Sep 11 22:00:00.432710 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.432679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"]
Sep 11 22:00:00.435222 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:00:00.435197    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd058e819_ff18_46da_b30d_7ee832fb7a34.slice/crio-4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c WatchSource:0}: Error finding container 4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c: Status 404 returned error can't find the container with id 4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c
Sep 11 22:00:00.444280 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.444258    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"]
Sep 11 22:00:00.447937 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:00:00.447913    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4530283_56e3_4628_8d0a_285b9f38db47.slice/crio-94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441 WatchSource:0}: Error finding container 94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441: Status 404 returned error can't find the container with id 94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441
Sep 11 22:00:00.491535 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.491505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g" event={"ID":"f4530283-56e3-4628-8d0a-285b9f38db47","Type":"ContainerStarted","Data":"94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441"}
Sep 11 22:00:00.492673 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:00.492646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n" event={"ID":"d058e819-ff18-46da-b30d-7ee832fb7a34","Type":"ContainerStarted","Data":"4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c"}
Sep 11 22:00:01.498787 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:01.498757    2582 generic.go:356] "Generic (PLEG): container finished" podID="d058e819-ff18-46da-b30d-7ee832fb7a34" containerID="29f3a34cbcde6e7c1ff94a060168cac2521824cbd29c0153a90456907a89e270" exitCode=0
Sep 11 22:00:01.499155 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:01.498847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n" event={"ID":"d058e819-ff18-46da-b30d-7ee832fb7a34","Type":"ContainerDied","Data":"29f3a34cbcde6e7c1ff94a060168cac2521824cbd29c0153a90456907a89e270"}
Sep 11 22:00:01.500476 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:01.500454    2582 generic.go:356] "Generic (PLEG): container finished" podID="f4530283-56e3-4628-8d0a-285b9f38db47" containerID="a7073c7c296bf425b6c01fa1d46d22bcd07f7ef7e6f762e60f15b8c06933c9b1" exitCode=0
Sep 11 22:00:01.500589 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:01.500484    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g" event={"ID":"f4530283-56e3-4628-8d0a-285b9f38db47","Type":"ContainerDied","Data":"a7073c7c296bf425b6c01fa1d46d22bcd07f7ef7e6f762e60f15b8c06933c9b1"}
Sep 11 22:00:02.656258 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:02.656230    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"
Sep 11 22:00:02.659110 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:02.659088    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"
Sep 11 22:00:03.511293 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.511267    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n"
Sep 11 22:00:03.511494 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.511314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819400-kgn4n" event={"ID":"d058e819-ff18-46da-b30d-7ee832fb7a34","Type":"ContainerDied","Data":"4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c"}
Sep 11 22:00:03.511494 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.511361    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f5e71bd0b4b0268c6f46693d501995f69aba88ad6f328b8a79d15dc79d1195c"
Sep 11 22:00:03.512686 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.512661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g" event={"ID":"f4530283-56e3-4628-8d0a-285b9f38db47","Type":"ContainerDied","Data":"94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441"}
Sep 11 22:00:03.512686 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.512684    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"
Sep 11 22:00:03.512865 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:03.512697    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94a9ab4a057b29f734e938671cdfaec231eee0206c72b2ade2a84ce5d1e5e441"
Sep 11 22:00:07.921130 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:07.921088    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:00:11.765069 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:00:11.765036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:01:00.788503 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:00.788439    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:01.158096 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:01.157994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:01:10.769526 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:10.769489    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:01:17.280684 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:17.280640    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:18.264908 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.264868    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:18.717578 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.717476    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:18.998415 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.998386    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:01:18.998922 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.998906    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d058e819-ff18-46da-b30d-7ee832fb7a34" containerName="cleanup"
Sep 11 22:01:18.998922 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.998924    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d058e819-ff18-46da-b30d-7ee832fb7a34" containerName="cleanup"
Sep 11 22:01:18.999090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.998943    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4530283-56e3-4628-8d0a-285b9f38db47" containerName="cleanup"
Sep 11 22:01:18.999090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.998948    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4530283-56e3-4628-8d0a-285b9f38db47" containerName="cleanup"
Sep 11 22:01:18.999090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.999019    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d058e819-ff18-46da-b30d-7ee832fb7a34" containerName="cleanup"
Sep 11 22:01:18.999090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:18.999026    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4530283-56e3-4628-8d0a-285b9f38db47" containerName="cleanup"
Sep 11 22:01:19.003011 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.002992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.005502 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.005484    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6a5690-openshift-default-dockercfg-vqrhf\""
Sep 11 22:01:19.013677 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.013655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:01:19.026848 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.026822    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:19.047496 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vck\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047632 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047632 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047632 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047804 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047804 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047804 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047939 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.047939 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.047888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148599 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148824 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148824 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148824 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148824 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.148824 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148849    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vck\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.148969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.149031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.149059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149349 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.149223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149385 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.149363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.149508 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.149490    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.151271 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.151249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.151602 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.151582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.156800 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.156780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.156911 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.156812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vck\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck\") pod \"e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") " pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.314822 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.314740    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:19.318084 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.318043    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:19.592204 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.592043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:01:19.595182 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:01:19.595156    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c14e05_9cb1_4be2_b713_21540e562c05.slice/crio-f711884f9c1f142210e5c673526aab907f2df95e43bc28b00c64b31779547c57 WatchSource:0}: Error finding container f711884f9c1f142210e5c673526aab907f2df95e43bc28b00c64b31779547c57: Status 404 returned error can't find the container with id f711884f9c1f142210e5c673526aab907f2df95e43bc28b00c64b31779547c57
Sep 11 22:01:19.598064 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.597997    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:19.598177 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.598101    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:19.598177 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.598145    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:19.902260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.902149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerStarted","Data":"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"}
Sep 11 22:01:19.902260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.902200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerStarted","Data":"f711884f9c1f142210e5c673526aab907f2df95e43bc28b00c64b31779547c57"}
Sep 11 22:01:19.921050 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:19.920994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podStartSLOduration=1.92097434 podStartE2EDuration="1.92097434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:19.919227811 +0000 UTC m=+1845.882252457" watchObservedRunningTime="2026-09-11 22:01:19.92097434 +0000 UTC m=+1845.883998991"
Sep 11 22:01:20.315757 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:20.315691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:20.320921 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:20.320894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:20.906941 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:20.906908    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:20.908007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:20.907983    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:21.066449 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:21.066413    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:01:21.103389 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:21.103358    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:21.883818 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:21.883771    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:21.980661 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:21.980617    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:22.886364 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:22.886329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:23.692320 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:23.692290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:23.751194 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:23.751162    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:23.890066 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:23.890031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:24.250910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.250880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:24.348292 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.348260    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:24.349158 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.349139    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:01:24.352970 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.352951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.356036 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.356006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6a5690-dockercfg-8gww8\""
Sep 11 22:01:24.356216 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.356189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6a5690\""
Sep 11 22:01:24.368249 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.368226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:01:24.389941 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.389906    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:01:24.393563 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.393544    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.403660 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.403628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:01:24.498795 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.498755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.498989 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.498805    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmc8\" (UniqueName: \"kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.498989 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.498872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtsl\" (UniqueName: \"kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.498989 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.498905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.600360 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.600270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.600360 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.600316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmc8\" (UniqueName: \"kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.600360 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.600352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtsl\" (UniqueName: \"kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.600586 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.600382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.601044 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.601025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.601164 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.601147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.609266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.609233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtsl\" (UniqueName: \"kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl\") pod \"payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.609391 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.609294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmc8\" (UniqueName: \"kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8\") pod \"payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.663130 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.663092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:24.704782 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.704744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:24.935061 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:24.934973    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:01:24.938961 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:01:24.938932    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9917af2_9dd7_4e4f_a871_e6d320fe78e6.slice/crio-fa614f42431ac8c232e3e83011cf7d58016993596f9cf6f622535b08b690ea5f WatchSource:0}: Error finding container fa614f42431ac8c232e3e83011cf7d58016993596f9cf6f622535b08b690ea5f: Status 404 returned error can't find the container with id fa614f42431ac8c232e3e83011cf7d58016993596f9cf6f622535b08b690ea5f
Sep 11 22:01:25.001323 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.001300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:01:25.003418 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:01:25.003378    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae949979_307d_4c4f_b04d_d808af837489.slice/crio-f8eace3499fc89a8ee65a3da460c3b4e7b427cbb03d91bb39994bb17dd2a3959 WatchSource:0}: Error finding container f8eace3499fc89a8ee65a3da460c3b4e7b427cbb03d91bb39994bb17dd2a3959: Status 404 returned error can't find the container with id f8eace3499fc89a8ee65a3da460c3b4e7b427cbb03d91bb39994bb17dd2a3959
Sep 11 22:01:25.936192 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.936142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" event={"ID":"ae949979-307d-4c4f-b04d-d808af837489","Type":"ContainerStarted","Data":"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"}
Sep 11 22:01:25.936680 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.936203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" event={"ID":"ae949979-307d-4c4f-b04d-d808af837489","Type":"ContainerStarted","Data":"f8eace3499fc89a8ee65a3da460c3b4e7b427cbb03d91bb39994bb17dd2a3959"}
Sep 11 22:01:25.936680 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.936357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:25.937820 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.937796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" event={"ID":"d9917af2-9dd7-4e4f-a871-e6d320fe78e6","Type":"ContainerStarted","Data":"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"}
Sep 11 22:01:25.937922 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.937836    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" event={"ID":"d9917af2-9dd7-4e4f-a871-e6d320fe78e6","Type":"ContainerStarted","Data":"fa614f42431ac8c232e3e83011cf7d58016993596f9cf6f622535b08b690ea5f"}
Sep 11 22:01:25.937976 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.937936    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:25.957389 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.957341    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" podStartSLOduration=1.957326343 podStartE2EDuration="1.957326343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:25.956216664 +0000 UTC m=+1851.919241327" watchObservedRunningTime="2026-09-11 22:01:25.957326343 +0000 UTC m=+1851.920350993"
Sep 11 22:01:25.976544 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:25.976499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" podStartSLOduration=1.976485155 podStartE2EDuration="1.976485155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:25.975468115 +0000 UTC m=+1851.938492764" watchObservedRunningTime="2026-09-11 22:01:25.976485155 +0000 UTC m=+1851.939509804"
Sep 11 22:01:30.294576 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:30.294531    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:33.570202 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:33.570155    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:01:36.945756 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:36.945705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:01:36.946260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:36.945814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:01:36.948925 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:36.948888    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 22:01:36.949068 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:36.948990    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 22:01:39.016050 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.016014    2582 generic.go:356] "Generic (PLEG): container finished" podID="44c14e05-9cb1-4be2-b713-21540e562c05" containerID="2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee" exitCode=0
Sep 11 22:01:39.016583 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.016093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerDied","Data":"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"}
Sep 11 22:01:39.016662 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.016632    2582 scope.go:117] "RemoveContainer" containerID="2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"
Sep 11 22:01:39.018160 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.018129    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:39.018260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.018196    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:39.018260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:39.018229    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:01:40.023708 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:40.023675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerStarted","Data":"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"}
Sep 11 22:01:40.315168 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:40.315062    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:40.315373 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:40.315349    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:01:40.315430 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:40.315411    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 22:01:41.316007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:41.315964    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:01:41.316426 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:41.316031    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 22:01:42.315152 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:42.315115    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:01:42.315348 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:42.315171    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 22:01:43.315313 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:43.315276    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:01:43.315765 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:43.315342    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 22:01:44.315193 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:44.315151    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:01:44.315407 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:44.315211    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 11 22:01:45.319475 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:45.319440    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:45.319969 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:45.319749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:45.320586 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:45.320569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:01:46.606910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:46.606873    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:46.859096 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:46.858994    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:47.341285 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:47.341252    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:47.569207 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:47.569170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:47.872234 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:47.872167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:48.046404 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:48.046349    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:48.265116 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:48.265069    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:49.067441 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:49.067386    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:49.982801 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:49.982760    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:50.706595 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:50.706518    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:50.804493 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:50.804449    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:51.703854 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:51.703817    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:52.428591 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:52.428560    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.059238 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.059197    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:01:53.062181 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.062152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.064510 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.064488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:01:53.064654 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.064541    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:01:53.065066 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.065046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-077193-serving-cert\""
Sep 11 22:01:53.065201 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.065077    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:01:53.065201 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.065138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2kttz\""
Sep 11 22:01:53.074294 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.074271    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:01:53.085019 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.084988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.085147 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.085091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.085147 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.085122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.085229 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.085208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fns9x\" (UniqueName: \"kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.097283 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.097259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.186402 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.186371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.186604 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.186415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.186604 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.186532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fns9x\" (UniqueName: \"kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.186708 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.186601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.187111 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.187092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.189036 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.189016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.189156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.189139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.194071 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.194045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fns9x\" (UniqueName: \"kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x\") pod \"maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.197133 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.197106    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.254226 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.254191    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.376642 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.376547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:53.596590 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.596551    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:53.649987 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.649960    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:01:53.651894 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:01:53.651866    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e595322_9288_4edd_8e98_abc5034bc4d5.slice/crio-f80c136782d8ae41e67ba1237cedd5787912fb713abd809f9b2af2aa17ba38b2 WatchSource:0}: Error finding container f80c136782d8ae41e67ba1237cedd5787912fb713abd809f9b2af2aa17ba38b2: Status 404 returned error can't find the container with id f80c136782d8ae41e67ba1237cedd5787912fb713abd809f9b2af2aa17ba38b2
Sep 11 22:01:53.653297 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.653278    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:01:53.685160 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:53.685117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:01:54.102260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:54.102232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" event={"ID":"3e595322-9288-4edd-8e98-abc5034bc4d5","Type":"ContainerStarted","Data":"1bcd6423aab79e15678bd3ffb5724156d399a3418809702a01e30f98b2d33d5a"}
Sep 11 22:01:54.102405 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:54.102289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" event={"ID":"3e595322-9288-4edd-8e98-abc5034bc4d5","Type":"ContainerStarted","Data":"f80c136782d8ae41e67ba1237cedd5787912fb713abd809f9b2af2aa17ba38b2"}
Sep 11 22:01:54.102503 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:54.102478    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:01:54.119440 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:54.119379    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" podStartSLOduration=0.739101474 podStartE2EDuration="1.119358128s" totalImagesPullingTime="380.256654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:54.116887143 +0000 UTC m=+1880.079911792" watchObservedRunningTime="2026-09-11 22:01:54.119358128 +0000 UTC m=+1880.082382778"
Sep 11 22:01:58.561045 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:01:58.560950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:02:00.113848 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:00.113818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:02:08.637195 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:08.637154    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:15.079426 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:15.079388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:15.649515 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:15.649481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:15.834539 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:15.834508    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:16.714339 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:16.714304    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:17.361917 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:17.361879    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:18.077397 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:18.077355    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:24.982400 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:24.982364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:27.524593 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:27.524561    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:29.256050 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:29.256014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:02:50.804636 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:50.804587    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:02:56.665032 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:56.665000    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn_ae949979-307d-4c4f-b04d-d808af837489/payload-processing/0.log"
Sep 11 22:02:57.208857 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.208809    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:57.463862 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.463749    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:57.515218 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.515179    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:02:57.518925 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.518906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.521358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.521331    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-3b9cac-openshift-default-dockercfg-nspv9\""
Sep 11 22:02:57.529828 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.529803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:02:57.607779 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.607779 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfn4\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607852    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.607980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.608005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608170 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.608026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.608170 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.608103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.709631 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.709631 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfn4\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709910    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.709960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.710045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.710225    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710589 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.710308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710589 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.710361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.710913 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.710889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.712242 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.712220    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.712378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.712360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.717192 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.717121    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.717192 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.717170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfn4\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4\") pod \"e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") " pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:57.781020 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.780977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:02:57.831147 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:57.831110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:58.111101 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.111069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:02:58.112877 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:02:58.112848    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4074ea60_fe50_4416_850d_caadd035fbc0.slice/crio-4308be4284dfee207b81f902388a739ea98847ed0d8fb4404fca79590add8cd8 WatchSource:0}: Error finding container 4308be4284dfee207b81f902388a739ea98847ed0d8fb4404fca79590add8cd8: Status 404 returned error can't find the container with id 4308be4284dfee207b81f902388a739ea98847ed0d8fb4404fca79590add8cd8
Sep 11 22:02:58.115295 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.115263    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:02:58.115372 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.115331    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:02:58.115372 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.115367    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:02:58.426630 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.426539    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerStarted","Data":"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"}
Sep 11 22:02:58.426630 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.426606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerStarted","Data":"4308be4284dfee207b81f902388a739ea98847ed0d8fb4404fca79590add8cd8"}
Sep 11 22:02:58.446934 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.446881    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podStartSLOduration=1.446864562 podStartE2EDuration="1.446864562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:02:58.445411528 +0000 UTC m=+1944.408436189" watchObservedRunningTime="2026-09-11 22:02:58.446864562 +0000 UTC m=+1944.409889211"
Sep 11 22:02:58.832181 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:58.832137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:02:59.838138 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:02:59.838109    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:00.175611 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.175520    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:00.369931 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.369884    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:00.441225 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.441136    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:00.442336 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.442312    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:00.469186 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.469145    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:00.981438 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:00.981400    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:01.804889 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:01.804845    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:02.417255 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:02.417208    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:03.160201 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.160166    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:03.260343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.260305    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:03.602067 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.602031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:03.697500 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.697461    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:03.706897 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.706871    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:03.710608 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.710591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.712839 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.712811    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3b9cac-dockercfg-lg2qh\""
Sep 11 22:03:03.712839 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.712837    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3b9cac\""
Sep 11 22:03:03.719460 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.719438    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:03.734902 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.734874    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:03.738361 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.738344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.747552 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.747531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:03.872557 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.872460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.872557 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.872505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h56z\" (UniqueName: \"kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.872557 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.872526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.872875 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.872592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62ng\" (UniqueName: \"kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.973217 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.973423 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h56z\" (UniqueName: \"kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.973423 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.973423 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62ng\" (UniqueName: \"kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.973948 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.974037 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.973983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:03.981456 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.981426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h56z\" (UniqueName: \"kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z\") pod \"payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:03.981592 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:03.981427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62ng\" (UniqueName: \"kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng\") pod \"payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:04.011146 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.011116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:04.022434 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.022405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:04.049516 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.049483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:04.303392 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.303368    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:04.305312 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:03:04.305285    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398d28f9_47e5_4f33_9a1f_78d339a319b2.slice/crio-0e1d2a17d866a171a0434367aca45d69d8f56ddb373e2372c87ce5571461598f WatchSource:0}: Error finding container 0e1d2a17d866a171a0434367aca45d69d8f56ddb373e2372c87ce5571461598f: Status 404 returned error can't find the container with id 0e1d2a17d866a171a0434367aca45d69d8f56ddb373e2372c87ce5571461598f
Sep 11 22:03:04.337212 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.337190    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:04.338931 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:03:04.338907    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7999b338_b0a1_42af_9ba9_00b8e87159b5.slice/crio-2b0aab2f9a416c5a017130d6624a098043bcebb4fc50310bcfa248b2d526fcbc WatchSource:0}: Error finding container 2b0aab2f9a416c5a017130d6624a098043bcebb4fc50310bcfa248b2d526fcbc: Status 404 returned error can't find the container with id 2b0aab2f9a416c5a017130d6624a098043bcebb4fc50310bcfa248b2d526fcbc
Sep 11 22:03:04.465773 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.465739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" event={"ID":"398d28f9-47e5-4f33-9a1f-78d339a319b2","Type":"ContainerStarted","Data":"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"}
Sep 11 22:03:04.465979 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.465897    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:04.465979 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.465956    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" event={"ID":"398d28f9-47e5-4f33-9a1f-78d339a319b2","Type":"ContainerStarted","Data":"0e1d2a17d866a171a0434367aca45d69d8f56ddb373e2372c87ce5571461598f"}
Sep 11 22:03:04.467348 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.467322    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" event={"ID":"7999b338-b0a1-42af-9ba9-00b8e87159b5","Type":"ContainerStarted","Data":"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"}
Sep 11 22:03:04.467489 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.467365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:04.467489 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.467380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" event={"ID":"7999b338-b0a1-42af-9ba9-00b8e87159b5","Type":"ContainerStarted","Data":"2b0aab2f9a416c5a017130d6624a098043bcebb4fc50310bcfa248b2d526fcbc"}
Sep 11 22:03:04.486153 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.486099    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" podStartSLOduration=1.4860798370000001 podStartE2EDuration="1.486079837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:04.483132276 +0000 UTC m=+1950.446156925" watchObservedRunningTime="2026-09-11 22:03:04.486079837 +0000 UTC m=+1950.449104489"
Sep 11 22:03:04.499946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.499892    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" podStartSLOduration=1.499875598 podStartE2EDuration="1.499875598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:04.497961879 +0000 UTC m=+1950.460986549" watchObservedRunningTime="2026-09-11 22:03:04.499875598 +0000 UTC m=+1950.462900243"
Sep 11 22:03:04.625831 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.625786    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:04.932591 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:04.932560    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:05.033284 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:05.033240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:10.064394 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:10.064354    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:03:15.475292 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:15.475257    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:15.475800 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:15.475392    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:20.558247 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.558156    2582 generic.go:356] "Generic (PLEG): container finished" podID="4074ea60-fe50-4416-850d-caadd035fbc0" containerID="b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4" exitCode=0
Sep 11 22:03:20.558247 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.558225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerDied","Data":"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"}
Sep 11 22:03:20.558696 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.558656    2582 scope.go:117] "RemoveContainer" containerID="b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"
Sep 11 22:03:20.559845 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.559814    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:03:20.559910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.559889    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:03:20.559945 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:20.559922    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 11 22:03:21.566330 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:21.566290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerStarted","Data":"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"}
Sep 11 22:03:21.831413 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:21.831317    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:21.831675 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:21.831649    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:03:21.831759 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:21.831705    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 22:03:22.831964 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:22.831917    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:03:22.832376 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:22.832002    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 22:03:23.832115 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:23.832082    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:03:23.832486 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:23.832136    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 22:03:24.832090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:24.832055    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:03:24.832285 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:24.832111    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 22:03:25.079445 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:25.079407    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:25.361076 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:25.361022    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:25.731194 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:25.731161    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:25.832029 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:25.831996    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:03:25.832201 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:25.832059    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 22:03:26.482647 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:26.482614    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:26.854582 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:26.854484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:26.854984 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:26.854951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:26.856990 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:26.856962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:27.100233 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:27.100196    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:27.861165 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:27.861130    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:28.509899 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:28.509856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:29.280000 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:29.279967    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:29.992776 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:29.992744    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:31.655578 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:31.655543    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.187590 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.187548    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.214140 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.214108    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:32.214526 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.214495    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" podUID="7999b338-b0a1-42af-9ba9-00b8e87159b5" containerName="payload-processing" containerID="cri-o://7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6" gracePeriod=30
Sep 11 22:03:32.226787 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.226748    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:32.233077 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.227238    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" podUID="398d28f9-47e5-4f33-9a1f-78d339a319b2" containerName="payload-pre-processing" containerID="cri-o://5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17" gracePeriod=30
Sep 11 22:03:32.255070 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.255032    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.312101 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.312053    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.376239 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.376205    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.431306 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.431270    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.521167 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.521140    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:32.547436 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.547410    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:32.631309 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.631277    2582 generic.go:356] "Generic (PLEG): container finished" podID="398d28f9-47e5-4f33-9a1f-78d339a319b2" containerID="5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17" exitCode=0
Sep 11 22:03:32.631477 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.631330    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"
Sep 11 22:03:32.631477 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.631359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" event={"ID":"398d28f9-47e5-4f33-9a1f-78d339a319b2","Type":"ContainerDied","Data":"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"}
Sep 11 22:03:32.631477 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.631416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v" event={"ID":"398d28f9-47e5-4f33-9a1f-78d339a319b2","Type":"ContainerDied","Data":"0e1d2a17d866a171a0434367aca45d69d8f56ddb373e2372c87ce5571461598f"}
Sep 11 22:03:32.631477 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.631446    2582 scope.go:117] "RemoveContainer" containerID="5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"
Sep 11 22:03:32.632633 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.632608    2582 generic.go:356] "Generic (PLEG): container finished" podID="7999b338-b0a1-42af-9ba9-00b8e87159b5" containerID="7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6" exitCode=0
Sep 11 22:03:32.632731 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.632655    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"
Sep 11 22:03:32.632731 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.632689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" event={"ID":"7999b338-b0a1-42af-9ba9-00b8e87159b5","Type":"ContainerDied","Data":"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"}
Sep 11 22:03:32.632841 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.632751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6" event={"ID":"7999b338-b0a1-42af-9ba9-00b8e87159b5","Type":"ContainerDied","Data":"2b0aab2f9a416c5a017130d6624a098043bcebb4fc50310bcfa248b2d526fcbc"}
Sep 11 22:03:32.640124 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.640108    2582 scope.go:117] "RemoveContainer" containerID="5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"
Sep 11 22:03:32.640363 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:32.640346    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17\": container with ID starting with 5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17 not found: ID does not exist" containerID="5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"
Sep 11 22:03:32.640411 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.640370    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17"} err="failed to get container status \"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17\": rpc error: code = NotFound desc = could not find container \"5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17\": container with ID starting with 5bdd12e764a01111a9e9bd6ab1324be565358d9fd12e00703832636c974a7e17 not found: ID does not exist"
Sep 11 22:03:32.640411 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.640387    2582 scope.go:117] "RemoveContainer" containerID="7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"
Sep 11 22:03:32.648801 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.648784    2582 scope.go:117] "RemoveContainer" containerID="7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"
Sep 11 22:03:32.649066 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:32.649048    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6\": container with ID starting with 7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6 not found: ID does not exist" containerID="7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"
Sep 11 22:03:32.649125 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.649077    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6"} err="failed to get container status \"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6\": rpc error: code = NotFound desc = could not find container \"7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6\": container with ID starting with 7b29e88331d58834ba195dc716c9deae6060863b4a160a346c38c50e1504f7e6 not found: ID does not exist"
Sep 11 22:03:32.677454 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.677425    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume\") pod \"398d28f9-47e5-4f33-9a1f-78d339a319b2\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") "
Sep 11 22:03:32.677857 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.677502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h56z\" (UniqueName: \"kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z\") pod \"398d28f9-47e5-4f33-9a1f-78d339a319b2\" (UID: \"398d28f9-47e5-4f33-9a1f-78d339a319b2\") "
Sep 11 22:03:32.677857 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.677552    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62ng\" (UniqueName: \"kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng\") pod \"7999b338-b0a1-42af-9ba9-00b8e87159b5\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") "
Sep 11 22:03:32.677857 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.677585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume\") pod \"7999b338-b0a1-42af-9ba9-00b8e87159b5\" (UID: \"7999b338-b0a1-42af-9ba9-00b8e87159b5\") "
Sep 11 22:03:32.678039 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.677882    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "398d28f9-47e5-4f33-9a1f-78d339a319b2" (UID: "398d28f9-47e5-4f33-9a1f-78d339a319b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:32.678039 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.678001    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/398d28f9-47e5-4f33-9a1f-78d339a319b2-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:32.678120 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.678036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7999b338-b0a1-42af-9ba9-00b8e87159b5" (UID: "7999b338-b0a1-42af-9ba9-00b8e87159b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:32.679796 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.679774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng" (OuterVolumeSpecName: "kube-api-access-n62ng") pod "7999b338-b0a1-42af-9ba9-00b8e87159b5" (UID: "7999b338-b0a1-42af-9ba9-00b8e87159b5"). InnerVolumeSpecName "kube-api-access-n62ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:32.679867 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.679822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z" (OuterVolumeSpecName: "kube-api-access-2h56z") pod "398d28f9-47e5-4f33-9a1f-78d339a319b2" (UID: "398d28f9-47e5-4f33-9a1f-78d339a319b2"). InnerVolumeSpecName "kube-api-access-2h56z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:32.747204 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.747170    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.778533 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.778502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h56z\" (UniqueName: \"kubernetes.io/projected/398d28f9-47e5-4f33-9a1f-78d339a319b2-kube-api-access-2h56z\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:32.778533 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.778530    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62ng\" (UniqueName: \"kubernetes.io/projected/7999b338-b0a1-42af-9ba9-00b8e87159b5-kube-api-access-n62ng\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:32.778533 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.778541    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7999b338-b0a1-42af-9ba9-00b8e87159b5-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:32.853665 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.853630    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:32.955514 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.955480    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:32.957423 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.957400    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3b9cac-fd64859b5-jfrx6"]
Sep 11 22:03:32.967076 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.967053    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:32.972840 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:32.972816    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3b9cac-565f76f4cd-28f8v"]
Sep 11 22:03:33.500397 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:33.500364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:34.167426 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:34.167391    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:34.455253 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:34.455178    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="398d28f9-47e5-4f33-9a1f-78d339a319b2" path="/var/lib/kubelet/pods/398d28f9-47e5-4f33-9a1f-78d339a319b2/volumes"
Sep 11 22:03:34.455569 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:34.455555    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7999b338-b0a1-42af-9ba9-00b8e87159b5" path="/var/lib/kubelet/pods/7999b338-b0a1-42af-9ba9-00b8e87159b5/volumes"
Sep 11 22:03:34.765795 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:34.765765    2582 scope.go:117] "RemoveContainer" containerID="4335475233faf0a1439ba29ed34955393c4c9e012597099508fa21ff2dbd4a10"
Sep 11 22:03:35.010132 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:35.010085    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:35.365850 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:35.365816    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:03:35.689931 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:35.689820    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:38.769918 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:38.769874    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:38.870105 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:38.870068    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:39.015067 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:39.015033    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:03:39.015315 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:39.015289    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" containerID="cri-o://9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2" gracePeriod=30
Sep 11 22:03:39.038449 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:39.038364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:41.857608 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:41.857564    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:03:41.858171 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:41.857628    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:03:42.958650 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:42.958616    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:43.687984 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:43.687933    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:44.274462 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.274437    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:44.290605 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290533    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290605 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfn4\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290707    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290783    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290837    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.290853 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290837    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:03:44.291178 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.290867    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo\") pod \"4074ea60-fe50-4416-850d-caadd035fbc0\" (UID: \"4074ea60-fe50-4416-850d-caadd035fbc0\") "
Sep 11 22:03:44.291178 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.291145    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.291665 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.291557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data" (OuterVolumeSpecName: "istio-data") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:03:44.291815 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.291661    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:03:44.291815 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.291682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:03:44.292301 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.292272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:44.293046 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.293013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token" (OuterVolumeSpecName: "istio-token") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:44.293948 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.293916    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4" (OuterVolumeSpecName: "kube-api-access-6lfn4") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "kube-api-access-6lfn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:44.293948 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.293925    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:03:44.294113 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.294064    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4074ea60-fe50-4416-850d-caadd035fbc0" (UID: "4074ea60-fe50-4416-850d-caadd035fbc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:03:44.327888 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.327857    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392445    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392486    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfn4\" (UniqueName: \"kubernetes.io/projected/4074ea60-fe50-4416-850d-caadd035fbc0-kube-api-access-6lfn4\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392517    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392532    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392545    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4074ea60-fe50-4416-850d-caadd035fbc0-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392558    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4074ea60-fe50-4416-850d-caadd035fbc0-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.392597 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.392571    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4074ea60-fe50-4416-850d-caadd035fbc0-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:44.699103 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.699008    2582 generic.go:356] "Generic (PLEG): container finished" podID="4074ea60-fe50-4416-850d-caadd035fbc0" containerID="9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2" exitCode=0
Sep 11 22:03:44.699103 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.699057    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"
Sep 11 22:03:44.699103 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.699093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerDied","Data":"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"}
Sep 11 22:03:44.699354 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.699149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx" event={"ID":"4074ea60-fe50-4416-850d-caadd035fbc0","Type":"ContainerDied","Data":"4308be4284dfee207b81f902388a739ea98847ed0d8fb4404fca79590add8cd8"}
Sep 11 22:03:44.699354 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.699172    2582 scope.go:117] "RemoveContainer" containerID="9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"
Sep 11 22:03:44.708281 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.708264    2582 scope.go:117] "RemoveContainer" containerID="b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"
Sep 11 22:03:44.715066 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.715043    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:03:44.717860 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.717822    2582 scope.go:117] "RemoveContainer" containerID="9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"
Sep 11 22:03:44.718183 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:44.718163    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2\": container with ID starting with 9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2 not found: ID does not exist" containerID="9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"
Sep 11 22:03:44.718247 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.718196    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2"} err="failed to get container status \"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2\": rpc error: code = NotFound desc = could not find container \"9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2\": container with ID starting with 9721c441946d3347d704a2b672055368ad5bf1f54e89af96eea5432c9373d9c2 not found: ID does not exist"
Sep 11 22:03:44.718247 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.718215    2582 scope.go:117] "RemoveContainer" containerID="b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"
Sep 11 22:03:44.718507 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:44.718489    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4\": container with ID starting with b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4 not found: ID does not exist" containerID="b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"
Sep 11 22:03:44.718549 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.718516    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4"} err="failed to get container status \"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4\": rpc error: code = NotFound desc = could not find container \"b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4\": container with ID starting with b6793423df0cd2c4f2e8c52545f56f5a046ee843a627d0e5f803913ae0cebca4 not found: ID does not exist"
Sep 11 22:03:44.718987 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:44.718969    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-3b9cac-openshift-default-6b6bc8766-ntkmx"]
Sep 11 22:03:45.089105 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:45.089066    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:45.890685 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:45.890654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:46.454990 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:46.454956    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" path="/var/lib/kubelet/pods/4074ea60-fe50-4416-850d-caadd035fbc0/volumes"
Sep 11 22:03:46.521017 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:46.520978    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:55.661725 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:55.661673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:03:58.184365 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.184323    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.211183 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.211150    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:03:58.211415 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.211389    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" podUID="ae949979-307d-4c4f-b04d-d808af837489" containerName="payload-processing" containerID="cri-o://018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff" gracePeriod=30
Sep 11 22:03:58.218475 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.218442    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:03:58.218739 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.218683    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" podUID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" containerName="payload-pre-processing" containerID="cri-o://a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f" gracePeriod=30
Sep 11 22:03:58.251841 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.251797    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.380904 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.380837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.467165 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.466737    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.517045 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.517006    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:03:58.520158 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.520123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:03:58.626787 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.626748    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume\") pod \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") "
Sep 11 22:03:58.626942 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.626800    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mmc8\" (UniqueName: \"kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8\") pod \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\" (UID: \"d9917af2-9dd7-4e4f-a871-e6d320fe78e6\") "
Sep 11 22:03:58.626942 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.626844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume\") pod \"ae949979-307d-4c4f-b04d-d808af837489\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") "
Sep 11 22:03:58.626942 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.626874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjtsl\" (UniqueName: \"kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl\") pod \"ae949979-307d-4c4f-b04d-d808af837489\" (UID: \"ae949979-307d-4c4f-b04d-d808af837489\") "
Sep 11 22:03:58.627225 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.627201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9917af2-9dd7-4e4f-a871-e6d320fe78e6" (UID: "d9917af2-9dd7-4e4f-a871-e6d320fe78e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:58.627262 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.627202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae949979-307d-4c4f-b04d-d808af837489" (UID: "ae949979-307d-4c4f-b04d-d808af837489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:03:58.629212 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.629191    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8" (OuterVolumeSpecName: "kube-api-access-9mmc8") pod "d9917af2-9dd7-4e4f-a871-e6d320fe78e6" (UID: "d9917af2-9dd7-4e4f-a871-e6d320fe78e6"). InnerVolumeSpecName "kube-api-access-9mmc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:58.629304 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.629219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl" (OuterVolumeSpecName: "kube-api-access-xjtsl") pod "ae949979-307d-4c4f-b04d-d808af837489" (UID: "ae949979-307d-4c4f-b04d-d808af837489"). InnerVolumeSpecName "kube-api-access-xjtsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:58.728073 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.728045    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mmc8\" (UniqueName: \"kubernetes.io/projected/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-kube-api-access-9mmc8\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:58.728073 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.728072    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae949979-307d-4c4f-b04d-d808af837489-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:58.728260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.728081    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjtsl\" (UniqueName: \"kubernetes.io/projected/ae949979-307d-4c4f-b04d-d808af837489-kube-api-access-xjtsl\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:58.728260 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.728090    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9917af2-9dd7-4e4f-a871-e6d320fe78e6-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:03:58.744160 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.744119    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.775962 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.775929    2582 generic.go:356] "Generic (PLEG): container finished" podID="ae949979-307d-4c4f-b04d-d808af837489" containerID="018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff" exitCode=0
Sep 11 22:03:58.776136 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.775978    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"
Sep 11 22:03:58.776136 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.776019    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" event={"ID":"ae949979-307d-4c4f-b04d-d808af837489","Type":"ContainerDied","Data":"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"}
Sep 11 22:03:58.776136 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.776078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn" event={"ID":"ae949979-307d-4c4f-b04d-d808af837489","Type":"ContainerDied","Data":"f8eace3499fc89a8ee65a3da460c3b4e7b427cbb03d91bb39994bb17dd2a3959"}
Sep 11 22:03:58.776136 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.776104    2582 scope.go:117] "RemoveContainer" containerID="018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"
Sep 11 22:03:58.777513 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.777490    2582 generic.go:356] "Generic (PLEG): container finished" podID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" containerID="a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f" exitCode=0
Sep 11 22:03:58.777602 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.777522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" event={"ID":"d9917af2-9dd7-4e4f-a871-e6d320fe78e6","Type":"ContainerDied","Data":"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"}
Sep 11 22:03:58.777602 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.777543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"
Sep 11 22:03:58.777602 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.777570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr" event={"ID":"d9917af2-9dd7-4e4f-a871-e6d320fe78e6","Type":"ContainerDied","Data":"fa614f42431ac8c232e3e83011cf7d58016993596f9cf6f622535b08b690ea5f"}
Sep 11 22:03:58.786268 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.786252    2582 scope.go:117] "RemoveContainer" containerID="018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"
Sep 11 22:03:58.786550 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:58.786531    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff\": container with ID starting with 018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff not found: ID does not exist" containerID="018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"
Sep 11 22:03:58.786636 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.786558    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff"} err="failed to get container status \"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff\": rpc error: code = NotFound desc = could not find container \"018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff\": container with ID starting with 018b96b76eab45413176189466d23a98ca9749335f5809143f7c03630c5601ff not found: ID does not exist"
Sep 11 22:03:58.786636 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.786574    2582 scope.go:117] "RemoveContainer" containerID="a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"
Sep 11 22:03:58.794826 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.794809    2582 scope.go:117] "RemoveContainer" containerID="a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"
Sep 11 22:03:58.795098 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:03:58.795080    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f\": container with ID starting with a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f not found: ID does not exist" containerID="a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"
Sep 11 22:03:58.795159 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.795107    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f"} err="failed to get container status \"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f\": rpc error: code = NotFound desc = could not find container \"a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f\": container with ID starting with a87f49acf004574bf1d7e3386935ed0207a78cf2cf62fe7089175efc7b10301f not found: ID does not exist"
Sep 11 22:03:58.799649 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.799626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:03:58.805028 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.805007    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a5690-6656787cbb-6h4sr"]
Sep 11 22:03:58.807419 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.807396    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:03:58.815368 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.815344    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:03:58.818745 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:58.818727    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a5690-78cff4d7c5-k9srn"]
Sep 11 22:03:59.447869 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:03:59.447837    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.073680 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:00.073646    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:00.454619 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:00.454526    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae949979-307d-4c4f-b04d-d808af837489" path="/var/lib/kubelet/pods/ae949979-307d-4c4f-b04d-d808af837489/volumes"
Sep 11 22:04:00.455116 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:00.455037    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" path="/var/lib/kubelet/pods/d9917af2-9dd7-4e4f-a871-e6d320fe78e6/volumes"
Sep 11 22:04:00.862294 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:00.862259    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:03.815782 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:03.815704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:08.967033 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:08.966996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:04:40.069811 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:40.069770    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:41.228122 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:41.228086    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:46.830887 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:46.830847    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:46.884307 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:46.884261    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:47.091666 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:47.091551    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:04:47.091930 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:47.091814    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy" containerID="cri-o://4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e" gracePeriod=30
Sep 11 22:04:47.113490 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:47.113447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:47.757952 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:47.757913    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:48.418058 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:48.418014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:49.183914 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:49.183870    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:49.821673 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:49.821638    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:50.578539 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:50.578498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:52.341539 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.341512    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:04:52.445083 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.444988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445083 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445047    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445161    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445183    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445216    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445244    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8vck\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445289    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket\") pod \"44c14e05-9cb1-4be2-b713-21540e562c05\" (UID: \"44c14e05-9cb1-4be2-b713-21540e562c05\") "
Sep 11 22:04:52.445701 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:52.445701 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445454    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:52.445701 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445485    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:52.445701 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445661    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:52.445918 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445704    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data" (OuterVolumeSpecName: "istio-data") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:52.445967 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445913    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.445967 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445935    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.445967 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445948    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.445967 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445962    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44c14e05-9cb1-4be2-b713-21540e562c05-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.446125 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.445975    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.447622 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.447595    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token" (OuterVolumeSpecName: "istio-token") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:52.447747 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.447627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:04:52.448029 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.448010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck" (OuterVolumeSpecName: "kube-api-access-d8vck") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "kube-api-access-d8vck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:52.448078 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.448018    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44c14e05-9cb1-4be2-b713-21540e562c05" (UID: "44c14e05-9cb1-4be2-b713-21540e562c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:04:52.546675 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.546635    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44c14e05-9cb1-4be2-b713-21540e562c05-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.546675 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.546669    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44c14e05-9cb1-4be2-b713-21540e562c05-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.546675 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.546680    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.546956 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:52.546692    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vck\" (UniqueName: \"kubernetes.io/projected/44c14e05-9cb1-4be2-b713-21540e562c05-kube-api-access-d8vck\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:53.051557 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.051524    2582 generic.go:356] "Generic (PLEG): container finished" podID="44c14e05-9cb1-4be2-b713-21540e562c05" containerID="4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e" exitCode=0
Sep 11 22:04:53.051774 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.051573    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"
Sep 11 22:04:53.051774 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.051613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerDied","Data":"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"}
Sep 11 22:04:53.051774 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.051670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7" event={"ID":"44c14e05-9cb1-4be2-b713-21540e562c05","Type":"ContainerDied","Data":"f711884f9c1f142210e5c673526aab907f2df95e43bc28b00c64b31779547c57"}
Sep 11 22:04:53.051774 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.051692    2582 scope.go:117] "RemoveContainer" containerID="4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"
Sep 11 22:04:53.060505 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.060486    2582 scope.go:117] "RemoveContainer" containerID="2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"
Sep 11 22:04:53.069697 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.069484    2582 scope.go:117] "RemoveContainer" containerID="4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"
Sep 11 22:04:53.069965 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:04:53.069838    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e\": container with ID starting with 4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e not found: ID does not exist" containerID="4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"
Sep 11 22:04:53.069965 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.069869    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e"} err="failed to get container status \"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e\": rpc error: code = NotFound desc = could not find container \"4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e\": container with ID starting with 4e05597f4f1cfe1d525f4075fd1c92d78d75fb28a5e4205fd225e2b7c0c29e5e not found: ID does not exist"
Sep 11 22:04:53.069965 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.069892    2582 scope.go:117] "RemoveContainer" containerID="2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"
Sep 11 22:04:53.070194 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:04:53.070169    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee\": container with ID starting with 2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee not found: ID does not exist" containerID="2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"
Sep 11 22:04:53.070241 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.070205    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee"} err="failed to get container status \"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee\": rpc error: code = NotFound desc = could not find container \"2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee\": container with ID starting with 2485e5abd2040ba8c97a0155bb40aae2971eb2daa89440efcc40f782b0c812ee not found: ID does not exist"
Sep 11 22:04:53.071023 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.071003    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:04:53.075924 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:53.075902    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a5690-openshift-default-957c86568-bb7v7"]
Sep 11 22:04:54.456338 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:54.456304    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" path="/var/lib/kubelet/pods/44c14e05-9cb1-4be2-b713-21540e562c05/volumes"
Sep 11 22:04:57.887595 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:57.887495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:04:57.888107 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:57.887790    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" podUID="3e595322-9288-4edd-8e98-abc5034bc4d5" containerName="maas-api" containerID="cri-o://1bcd6423aab79e15678bd3ffb5724156d399a3418809702a01e30f98b2d33d5a" gracePeriod=30
Sep 11 22:04:57.939640 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:57.939598    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:58.038703 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.038660    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:58.085852 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.085823    2582 generic.go:356] "Generic (PLEG): container finished" podID="3e595322-9288-4edd-8e98-abc5034bc4d5" containerID="1bcd6423aab79e15678bd3ffb5724156d399a3418809702a01e30f98b2d33d5a" exitCode=0
Sep 11 22:04:58.085994 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.085894    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" event={"ID":"3e595322-9288-4edd-8e98-abc5034bc4d5","Type":"ContainerDied","Data":"1bcd6423aab79e15678bd3ffb5724156d399a3418809702a01e30f98b2d33d5a"}
Sep 11 22:04:58.113353 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.113318    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:58.139931 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.139871    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:04:58.301360 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.301328    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls\") pod \"3e595322-9288-4edd-8e98-abc5034bc4d5\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") "
Sep 11 22:04:58.301538 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.301400    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs\") pod \"3e595322-9288-4edd-8e98-abc5034bc4d5\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") "
Sep 11 22:04:58.301538 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.301438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fns9x\" (UniqueName: \"kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x\") pod \"3e595322-9288-4edd-8e98-abc5034bc4d5\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") "
Sep 11 22:04:58.301538 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.301463    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca\") pod \"3e595322-9288-4edd-8e98-abc5034bc4d5\" (UID: \"3e595322-9288-4edd-8e98-abc5034bc4d5\") "
Sep 11 22:04:58.301931 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.301899    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e595322-9288-4edd-8e98-abc5034bc4d5" (UID: "3e595322-9288-4edd-8e98-abc5034bc4d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:58.303779 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.303747    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e595322-9288-4edd-8e98-abc5034bc4d5" (UID: "3e595322-9288-4edd-8e98-abc5034bc4d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:04:58.303779 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.303768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x" (OuterVolumeSpecName: "kube-api-access-fns9x") pod "3e595322-9288-4edd-8e98-abc5034bc4d5" (UID: "3e595322-9288-4edd-8e98-abc5034bc4d5"). InnerVolumeSpecName "kube-api-access-fns9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:58.303897 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.303859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e595322-9288-4edd-8e98-abc5034bc4d5" (UID: "3e595322-9288-4edd-8e98-abc5034bc4d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:04:58.403156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.403069    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.403156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.403100    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fns9x\" (UniqueName: \"kubernetes.io/projected/3e595322-9288-4edd-8e98-abc5034bc4d5-kube-api-access-fns9x\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.403156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.403109    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e595322-9288-4edd-8e98-abc5034bc4d5-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.403156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.403118    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e595322-9288-4edd-8e98-abc5034bc4d5-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:04:58.521443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.521399    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:58.621463 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:58.621408    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:04:59.091890 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.091859    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"
Sep 11 22:04:59.092367 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.091890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz" event={"ID":"3e595322-9288-4edd-8e98-abc5034bc4d5","Type":"ContainerDied","Data":"f80c136782d8ae41e67ba1237cedd5787912fb713abd809f9b2af2aa17ba38b2"}
Sep 11 22:04:59.092367 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.091948    2582 scope.go:117] "RemoveContainer" containerID="1bcd6423aab79e15678bd3ffb5724156d399a3418809702a01e30f98b2d33d5a"
Sep 11 22:04:59.110050 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.110015    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:04:59.114467 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.114440    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-077193-7c594b9c78-tdvpz"]
Sep 11 22:04:59.411335 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:04:59.411237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:00.348548 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:00.348505    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:00.454292 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:00.454255    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e595322-9288-4edd-8e98-abc5034bc4d5" path="/var/lib/kubelet/pods/3e595322-9288-4edd-8e98-abc5034bc4d5/volumes"
Sep 11 22:05:01.172633 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:01.172588    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:05:04.180289 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.180256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:04.281002 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.280968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:04.446896 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.446788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:04.799641 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.799599    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"]
Sep 11 22:05:04.800005 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.799991    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7999b338-b0a1-42af-9ba9-00b8e87159b5" containerName="payload-processing"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800007    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7999b338-b0a1-42af-9ba9-00b8e87159b5" containerName="payload-processing"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800024    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e595322-9288-4edd-8e98-abc5034bc4d5" containerName="maas-api"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800032    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e595322-9288-4edd-8e98-abc5034bc4d5" containerName="maas-api"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800042    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800048    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800061    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800066    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800079    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" containerName="payload-pre-processing"
Sep 11 22:05:04.800081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800084    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" containerName="payload-pre-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800090    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="398d28f9-47e5-4f33-9a1f-78d339a319b2" containerName="payload-pre-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800094    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="398d28f9-47e5-4f33-9a1f-78d339a319b2" containerName="payload-pre-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800102    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae949979-307d-4c4f-b04d-d808af837489" containerName="payload-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800106    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae949979-307d-4c4f-b04d-d808af837489" containerName="payload-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800166    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800173    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae949979-307d-4c4f-b04d-d808af837489" containerName="payload-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800180    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800186    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800191    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e595322-9288-4edd-8e98-abc5034bc4d5" containerName="maas-api"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800198    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9917af2-9dd7-4e4f-a871-e6d320fe78e6" containerName="payload-pre-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800204    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="398d28f9-47e5-4f33-9a1f-78d339a319b2" containerName="payload-pre-processing"
Sep 11 22:05:04.800358 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.800210    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7999b338-b0a1-42af-9ba9-00b8e87159b5" containerName="payload-processing"
Sep 11 22:05:04.804658 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.804637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:04.806938 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.806905    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:05:04.807151 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.807136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:05:04.813808 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.813779    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"]
Sep 11 22:05:04.857911 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.857876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2\" (UID: \"48a3d330-35cd-4af7-a20f-82cfc6baee24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:04.905062 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.905025    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:04.958521 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.958476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2\" (UID: \"48a3d330-35cd-4af7-a20f-82cfc6baee24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:04.959170 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:04.959150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2\" (UID: \"48a3d330-35cd-4af7-a20f-82cfc6baee24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:05.118144 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:05.118057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:05.387101 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:05.387076    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"]
Sep 11 22:05:05.390067 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:05:05.390033    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a3d330_35cd_4af7_a20f_82cfc6baee24.slice/crio-d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326 WatchSource:0}: Error finding container d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326: Status 404 returned error can't find the container with id d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326
Sep 11 22:05:05.707479 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:05.707380    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:06.128774 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:06.128744    2582 generic.go:356] "Generic (PLEG): container finished" podID="48a3d330-35cd-4af7-a20f-82cfc6baee24" containerID="f4446b9da1aad678481329605f196248a7a1c25a43d0d6a38f2f582fb9307e96" exitCode=0
Sep 11 22:05:06.128947 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:06.128796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2" event={"ID":"48a3d330-35cd-4af7-a20f-82cfc6baee24","Type":"ContainerDied","Data":"f4446b9da1aad678481329605f196248a7a1c25a43d0d6a38f2f582fb9307e96"}
Sep 11 22:05:06.128947 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:06.128846    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2" event={"ID":"48a3d330-35cd-4af7-a20f-82cfc6baee24","Type":"ContainerStarted","Data":"d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326"}
Sep 11 22:05:06.491507 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:06.491464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:07.174082 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:07.174015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:07.276057 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:07.276028    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:07.379520 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:07.379485    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca\") pod \"48a3d330-35cd-4af7-a20f-82cfc6baee24\" (UID: \"48a3d330-35cd-4af7-a20f-82cfc6baee24\") "
Sep 11 22:05:07.379888 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:07.379865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48a3d330-35cd-4af7-a20f-82cfc6baee24" (UID: "48a3d330-35cd-4af7-a20f-82cfc6baee24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:07.480910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:07.480866    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48a3d330-35cd-4af7-a20f-82cfc6baee24-maas-api-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:08.140216 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:08.140190    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"
Sep 11 22:05:08.140699 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:08.140228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2" event={"ID":"48a3d330-35cd-4af7-a20f-82cfc6baee24","Type":"ContainerDied","Data":"d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326"}
Sep 11 22:05:08.140699 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:08.140276    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d694bd949ca5f8bcd2881f7be39fc6ace2da75351abbf328214224ba3ad4f326"
Sep 11 22:05:15.411183 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.411141    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:15.443333 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.443295    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 22:05:15.443598 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.443554    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" podUID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" containerName="payload-pre-processing" containerID="cri-o://53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df" gracePeriod=30
Sep 11 22:05:15.509292 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.509256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:15.589156 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.589086    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"]
Sep 11 22:05:15.591049 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.591019    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7d7821-ndgc2"]
Sep 11 22:05:15.591532 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.591511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:15.649523 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.649492    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:15.717692 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.717659    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 22:05:15.761328 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.761296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dg2g\" (UniqueName: \"kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g\") pod \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") "
Sep 11 22:05:15.761328 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.761331    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume\") pod \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\" (UID: \"db559bf1-c1a7-40d7-beba-6cbe1931e4bb\") "
Sep 11 22:05:15.761825 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.761795    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db559bf1-c1a7-40d7-beba-6cbe1931e4bb" (UID: "db559bf1-c1a7-40d7-beba-6cbe1931e4bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:15.763653 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.763623    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g" (OuterVolumeSpecName: "kube-api-access-8dg2g") pod "db559bf1-c1a7-40d7-beba-6cbe1931e4bb" (UID: "db559bf1-c1a7-40d7-beba-6cbe1931e4bb"). InnerVolumeSpecName "kube-api-access-8dg2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:15.862644 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.862598    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dg2g\" (UniqueName: \"kubernetes.io/projected/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-kube-api-access-8dg2g\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:15.862644 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:15.862633    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db559bf1-c1a7-40d7-beba-6cbe1931e4bb-plugins-config-volume\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:16.001210 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.001168    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:16.085216 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.085171    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:16.181780 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.181741    2582 generic.go:356] "Generic (PLEG): container finished" podID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" containerID="53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df" exitCode=0
Sep 11 22:05:16.181780 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.181771    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"
Sep 11 22:05:16.182040 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.181815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" event={"ID":"db559bf1-c1a7-40d7-beba-6cbe1931e4bb","Type":"ContainerDied","Data":"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"}
Sep 11 22:05:16.182040 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.181875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5" event={"ID":"db559bf1-c1a7-40d7-beba-6cbe1931e4bb","Type":"ContainerDied","Data":"1e0338fd36fbcc20bffe36ee6f04f0d1fd6a04357b54e2bba3dd1c766e487dcb"}
Sep 11 22:05:16.182040 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.181902    2582 scope.go:117] "RemoveContainer" containerID="53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"
Sep 11 22:05:16.191098 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.191080    2582 scope.go:117] "RemoveContainer" containerID="53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"
Sep 11 22:05:16.191443 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:05:16.191421    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df\": container with ID starting with 53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df not found: ID does not exist" containerID="53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"
Sep 11 22:05:16.191493 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.191455    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df"} err="failed to get container status \"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df\": rpc error: code = NotFound desc = could not find container \"53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df\": container with ID starting with 53ced432f58990476267de52378173326daa5b8ce90abf2d5015d64e74ce97df not found: ID does not exist"
Sep 11 22:05:16.206259 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.206223    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 22:05:16.207877 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.207856    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7d7821-6688c4fd78-vffm5"]
Sep 11 22:05:16.454679 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.454600    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a3d330-35cd-4af7-a20f-82cfc6baee24" path="/var/lib/kubelet/pods/48a3d330-35cd-4af7-a20f-82cfc6baee24/volumes"
Sep 11 22:05:16.455056 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.455030    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" path="/var/lib/kubelet/pods/db559bf1-c1a7-40d7-beba-6cbe1931e4bb/volumes"
Sep 11 22:05:16.681364 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:16.681329    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:18.355412 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:18.355372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:05:21.693933 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:21.693891    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:21.794957 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:21.794921    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:21.935609 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:21.935576    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 22:05:21.935855 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:21.935828    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy" containerID="cri-o://ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc" gracePeriod=30
Sep 11 22:05:21.956074 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:21.955968    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:22.287875 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.287829    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"]
Sep 11 22:05:22.288285 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288266    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288288    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4074ea60-fe50-4416-850d-caadd035fbc0" containerName="istio-proxy"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288300    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a3d330-35cd-4af7-a20f-82cfc6baee24" containerName="revoke-keys"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288308    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a3d330-35cd-4af7-a20f-82cfc6baee24" containerName="revoke-keys"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288323    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288331    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288344    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" containerName="payload-pre-processing"
Sep 11 22:05:22.288378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288353    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" containerName="payload-pre-processing"
Sep 11 22:05:22.288781 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288444    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c14e05-9cb1-4be2-b713-21540e562c05" containerName="istio-proxy"
Sep 11 22:05:22.288781 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288463    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a3d330-35cd-4af7-a20f-82cfc6baee24" containerName="revoke-keys"
Sep 11 22:05:22.288781 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.288473    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="db559bf1-c1a7-40d7-beba-6cbe1931e4bb" containerName="payload-pre-processing"
Sep 11 22:05:22.293143 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.293119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:22.295283 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.295260    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:05:22.295660 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.295264    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rwgd\""
Sep 11 22:05:22.298082 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.298048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"]
Sep 11 22:05:22.429568 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.429526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb\" (UID: \"f5d235dd-c373-4721-928c-f5daf4082db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:22.530949 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.530914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb\" (UID: \"f5d235dd-c373-4721-928c-f5daf4082db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:22.531551 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.531530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb\" (UID: \"f5d235dd-c373-4721-928c-f5daf4082db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:22.568513 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.568414    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:22.605406 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.605368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:22.879138 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:22.879114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"]
Sep 11 22:05:22.881503 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:05:22.881477    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d235dd_c373_4721_928c_f5daf4082db1.slice/crio-b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e WatchSource:0}: Error finding container b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e: Status 404 returned error can't find the container with id b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e
Sep 11 22:05:23.225907 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:23.225868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb" event={"ID":"f5d235dd-c373-4721-928c-f5daf4082db1","Type":"ContainerStarted","Data":"5351d722709512196ba2b291feb699e1838ce6cb8b519d66bc40116645c77494"}
Sep 11 22:05:23.226087 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:23.225930    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb" event={"ID":"f5d235dd-c373-4721-928c-f5daf4082db1","Type":"ContainerStarted","Data":"b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e"}
Sep 11 22:05:23.240451 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:23.240402    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb" podStartSLOduration=1.240386408 podStartE2EDuration="1.240386408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:23.238946197 +0000 UTC m=+2089.201970852" watchObservedRunningTime="2026-09-11 22:05:23.240386408 +0000 UTC m=+2089.203411057"
Sep 11 22:05:23.243376 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:23.243346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:23.891705 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:23.891607    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:24.232221 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:24.232191    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5d235dd-c373-4721-928c-f5daf4082db1" containerID="5351d722709512196ba2b291feb699e1838ce6cb8b519d66bc40116645c77494" exitCode=0
Sep 11 22:05:24.232421 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:24.232252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb" event={"ID":"f5d235dd-c373-4721-928c-f5daf4082db1","Type":"ContainerDied","Data":"5351d722709512196ba2b291feb699e1838ce6cb8b519d66bc40116645c77494"}
Sep 11 22:05:25.367811 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:25.367781    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:25.458031 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:25.457996    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca\") pod \"f5d235dd-c373-4721-928c-f5daf4082db1\" (UID: \"f5d235dd-c373-4721-928c-f5daf4082db1\") "
Sep 11 22:05:25.458347 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:25.458321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5d235dd-c373-4721-928c-f5daf4082db1" (UID: "f5d235dd-c373-4721-928c-f5daf4082db1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:25.559811 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:25.559692    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d235dd-c373-4721-928c-f5daf4082db1-maas-api-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:26.242975 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:26.242943    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"
Sep 11 22:05:26.243157 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:26.242984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb" event={"ID":"f5d235dd-c373-4721-928c-f5daf4082db1","Type":"ContainerDied","Data":"b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e"}
Sep 11 22:05:26.243157 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:26.243035    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1758a7cf9e3128ece23f11194d78311d3ce1617218121996f261ec84f8b514e"
Sep 11 22:05:27.191730 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.191690    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 22:05:27.249639 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.249603    2582 generic.go:356] "Generic (PLEG): container finished" podID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerID="ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc" exitCode=0
Sep 11 22:05:27.249835 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.249653    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"
Sep 11 22:05:27.249835 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.249687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerDied","Data":"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"}
Sep 11 22:05:27.249835 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.249766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s" event={"ID":"e8bfa079-a16b-4b05-b52d-ffdeac8cca03","Type":"ContainerDied","Data":"344b7f8b5d4ddfd5cdbd2c3a8321b48f86d075a3ff2adf4a5452b58cde3889bd"}
Sep 11 22:05:27.249835 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.249792    2582 scope.go:117] "RemoveContainer" containerID="ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"
Sep 11 22:05:27.258815 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.258797    2582 scope.go:117] "RemoveContainer" containerID="ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"
Sep 11 22:05:27.267278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.267260    2582 scope.go:117] "RemoveContainer" containerID="ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"
Sep 11 22:05:27.267550 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:05:27.267533    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc\": container with ID starting with ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc not found: ID does not exist" containerID="ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"
Sep 11 22:05:27.267600 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.267559    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc"} err="failed to get container status \"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc\": rpc error: code = NotFound desc = could not find container \"ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc\": container with ID starting with ef11d82311eec07b50222d91139bf1c71ee35ba7f1fbb53eafaa95c2d0e2cedc not found: ID does not exist"
Sep 11 22:05:27.267600 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.267577    2582 scope.go:117] "RemoveContainer" containerID="ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"
Sep 11 22:05:27.267836 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:05:27.267817    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482\": container with ID starting with ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482 not found: ID does not exist" containerID="ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"
Sep 11 22:05:27.267898 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.267846    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482"} err="failed to get container status \"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482\": rpc error: code = NotFound desc = could not find container \"ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482\": container with ID starting with ee78c4e3b631e2eec3c48e6853872069d5a0db4c4ef063aee1dda2e2c6b10482 not found: ID does not exist"
Sep 11 22:05:27.278203 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278180    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278289 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278289 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278378 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278467 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx6jv\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278467 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278467 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278457    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278610 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278494    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data\") pod \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\" (UID: \"e8bfa079-a16b-4b05-b52d-ffdeac8cca03\") "
Sep 11 22:05:27.278610 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:27.278781 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278635    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:27.278846 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278795    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:27.278846 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278814    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:27.278963 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:27.278963 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278934    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.278963 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278953    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.279072 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278966    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.279072 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.278979    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.280576 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.280553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:05:27.280697 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.280668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv" (OuterVolumeSpecName: "kube-api-access-qx6jv") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "kube-api-access-qx6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:27.281099 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.281077    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:27.281176 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.281089    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8bfa079-a16b-4b05-b52d-ffdeac8cca03" (UID: "e8bfa079-a16b-4b05-b52d-ffdeac8cca03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:27.379667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.379629    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.379667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.379660    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.379667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.379670    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx6jv\" (UniqueName: \"kubernetes.io/projected/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-kube-api-access-qx6jv\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.379919 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.379680    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.379919 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.379688    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bfa079-a16b-4b05-b52d-ffdeac8cca03-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.572226 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.572187    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 22:05:27.576830 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:27.576801    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-7d7821-openshift-default-5c97dd94db-wm46s"]
Sep 11 22:05:28.454649 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:28.454616    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" path="/var/lib/kubelet/pods/e8bfa079-a16b-4b05-b52d-ffdeac8cca03/volumes"
Sep 11 22:05:29.827571 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:29.827535    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.829145 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:32.828905    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.831087 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:32.831066    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"]
Sep 11 22:05:32.833766 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:32.833742    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4e0323-29819400-m6x8g"]
Sep 11 22:05:32.902058 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:32.902015    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:32.961480 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:32.961415    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.008983 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:33.008930    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"]
Sep 11 22:05:33.010343 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:33.010323    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4e0323-vkfzb"]
Sep 11 22:05:33.025749 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:33.025704    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.457168 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:33.457137    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:33.556325 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:33.556290    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:34.454927 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:34.454893    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4530283-56e3-4628-8d0a-285b9f38db47" path="/var/lib/kubelet/pods/f4530283-56e3-4628-8d0a-285b9f38db47/volumes"
Sep 11 22:05:34.455304 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:34.455188    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d235dd-c373-4721-928c-f5daf4082db1" path="/var/lib/kubelet/pods/f5d235dd-c373-4721-928c-f5daf4082db1/volumes"
Sep 11 22:05:35.764889 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:35.764853    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:05:39.299648 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:39.299562    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:39.365693 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:39.365642    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:39.553677 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:39.553593    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 22:05:39.553872 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:39.553848    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy" containerID="cri-o://e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4" gracePeriod=30
Sep 11 22:05:39.579402 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:39.579366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:44.135977 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.135935    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:44.237075 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.237040    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:05:44.801308 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.801281    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 22:05:44.837667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837587    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837667 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837644    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837686    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dtn4\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837740    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837806    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837844    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.837946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.837904    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token\") pod \"001fbf53-831b-4603-9fb3-2f2c3d608c53\" (UID: \"001fbf53-831b-4603-9fb3-2f2c3d608c53\") "
Sep 11 22:05:44.838255 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838142    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:44.838310 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838286    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.838708 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:44.838708 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838684    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:44.838890 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:44.838976 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.838958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data" (OuterVolumeSpecName: "istio-data") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:44.840502 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.840456    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:05:44.840634 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.840616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:05:44.841020 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.840991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token" (OuterVolumeSpecName: "istio-token") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:44.841365 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.841331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4" (OuterVolumeSpecName: "kube-api-access-6dtn4") pod "001fbf53-831b-4603-9fb3-2f2c3d608c53" (UID: "001fbf53-831b-4603-9fb3-2f2c3d608c53"). InnerVolumeSpecName "kube-api-access-6dtn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:44.939577 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939540    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-token\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939577 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939571    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-podinfo\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939577 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939580    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-envoy\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939577 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939589    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-istio-data\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939907 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939598    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtn4\" (UniqueName: \"kubernetes.io/projected/001fbf53-831b-4603-9fb3-2f2c3d608c53-kube-api-access-6dtn4\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939907 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939608    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001fbf53-831b-4603-9fb3-2f2c3d608c53-istiod-ca-cert\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939907 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939616    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-credential-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.939907 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.939624    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001fbf53-831b-4603-9fb3-2f2c3d608c53-workload-socket\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:05:44.966245 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:44.966204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:05:45.342779 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.342747    2582 generic.go:356] "Generic (PLEG): container finished" podID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerID="e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4" exitCode=0
Sep 11 22:05:45.343288 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.342816    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerDied","Data":"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"}
Sep 11 22:05:45.343288 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.342819    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"
Sep 11 22:05:45.343288 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.342883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4" event={"ID":"001fbf53-831b-4603-9fb3-2f2c3d608c53","Type":"ContainerDied","Data":"cd08edb6ff9a4d59017ee7c81f8bb54f9671ac19efaa5efe2aa790ce5e8f1238"}
Sep 11 22:05:45.343288 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.342905    2582 scope.go:117] "RemoveContainer" containerID="e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"
Sep 11 22:05:45.352349 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.352333    2582 scope.go:117] "RemoveContainer" containerID="e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"
Sep 11 22:05:45.363381 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.363350    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 22:05:45.363689 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.363668    2582 scope.go:117] "RemoveContainer" containerID="e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"
Sep 11 22:05:45.364070 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:05:45.364043    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4\": container with ID starting with e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4 not found: ID does not exist" containerID="e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"
Sep 11 22:05:45.364132 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.364080    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4"} err="failed to get container status \"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4\": rpc error: code = NotFound desc = could not find container \"e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4\": container with ID starting with e1c9f958b57d94e6bb88e918fd65a90e773e40b031530d75c3c212a7451ac2f4 not found: ID does not exist"
Sep 11 22:05:45.364132 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.364101    2582 scope.go:117] "RemoveContainer" containerID="e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"
Sep 11 22:05:45.364488 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:05:45.364462    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9\": container with ID starting with e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9 not found: ID does not exist" containerID="e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"
Sep 11 22:05:45.364618 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.364490    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9"} err="failed to get container status \"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9\": rpc error: code = NotFound desc = could not find container \"e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9\": container with ID starting with e238578da755671f33acb1e765d14211a8b413911b98f1542d9e7d5b21917ee9 not found: ID does not exist"
Sep 11 22:05:45.367107 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:45.367084    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e0323-openshift-default-6db4bb5648-wz8t4"]
Sep 11 22:05:46.454507 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:46.454472    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" path="/var/lib/kubelet/pods/001fbf53-831b-4603-9fb3-2f2c3d608c53/volumes"
Sep 11 22:05:51.186699 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:05:51.186663    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:06:34.847759 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:06:34.847653    2582 scope.go:117] "RemoveContainer" containerID="a7073c7c296bf425b6c01fa1d46d22bcd07f7ef7e6f762e60f15b8c06933c9b1"
Sep 11 22:07:07.159780 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:07.159738    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:07:07.991678 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:07.991637    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:09.541044 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541003    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f59564f6b-p7nlw"]
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541374    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541386    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541400    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541408    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541428    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d235dd-c373-4721-928c-f5daf4082db1" containerName="revoke-keys"
Sep 11 22:07:09.541443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541434    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d235dd-c373-4721-928c-f5daf4082db1" containerName="revoke-keys"
Sep 11 22:07:09.541638 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541496    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d235dd-c373-4721-928c-f5daf4082db1" containerName="revoke-keys"
Sep 11 22:07:09.541638 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541504    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:07:09.541638 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541512    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:07:09.541638 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.541519    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:07:09.544587 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.544567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.547385 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.547355    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 22:07:09.547528 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.547355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wcldp\""
Sep 11 22:07:09.547528 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.547358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 22:07:09.547528 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.547403    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 22:07:09.555105 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.555081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f59564f6b-p7nlw"]
Sep 11 22:07:09.557578 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.557541    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:07:09.663744 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.663681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-webhook-cert\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.663946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.663816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-metrics-certs\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.663946 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.663847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzqcb\" (UniqueName: \"kubernetes.io/projected/7a42313a-c78f-4201-b684-02867ae09065-kube-api-access-xzqcb\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.765268 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.765233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-metrics-certs\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.765268 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.765278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzqcb\" (UniqueName: \"kubernetes.io/projected/7a42313a-c78f-4201-b684-02867ae09065-kube-api-access-xzqcb\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.765499 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.765351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-webhook-cert\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.768196 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.768160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-webhook-cert\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.768196 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.768160    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a42313a-c78f-4201-b684-02867ae09065-metrics-certs\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.772672 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.772642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzqcb\" (UniqueName: \"kubernetes.io/projected/7a42313a-c78f-4201-b684-02867ae09065-kube-api-access-xzqcb\") pod \"maas-controller-5f59564f6b-p7nlw\" (UID: \"7a42313a-c78f-4201-b684-02867ae09065\") " pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:09.855930 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:09.855841    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:10.128370 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:10.128329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f59564f6b-p7nlw"]
Sep 11 22:07:10.130561 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:07:10.130532    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a42313a_c78f_4201_b684_02867ae09065.slice/crio-e0c9271202933e2363f928046996e4877ba9cb796e67da5c38a7503be4875935 WatchSource:0}: Error finding container e0c9271202933e2363f928046996e4877ba9cb796e67da5c38a7503be4875935: Status 404 returned error can't find the container with id e0c9271202933e2363f928046996e4877ba9cb796e67da5c38a7503be4875935
Sep 11 22:07:10.131864 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:10.131846    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:07:10.756461 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:10.756421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f59564f6b-p7nlw" event={"ID":"7a42313a-c78f-4201-b684-02867ae09065","Type":"ContainerStarted","Data":"e0c9271202933e2363f928046996e4877ba9cb796e67da5c38a7503be4875935"}
Sep 11 22:07:12.767383 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:12.767347    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f59564f6b-p7nlw" event={"ID":"7a42313a-c78f-4201-b684-02867ae09065","Type":"ContainerStarted","Data":"ba8c51c967a9861aab186c9460b4c341b693af7dd9635e84ec38afd0006ffa82"}
Sep 11 22:07:12.767896 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:12.767407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:12.785556 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:12.785508    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f59564f6b-p7nlw" podStartSLOduration=2.115505887 podStartE2EDuration="3.785493759s" totalImagesPullingTime="1.669987872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:12.783920526 +0000 UTC m=+2198.746945175" watchObservedRunningTime="2026-09-11 22:07:12.785493759 +0000 UTC m=+2198.748518414"
Sep 11 22:07:23.775106 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:23.775069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f59564f6b-p7nlw"
Sep 11 22:07:35.755308 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:35.755274    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:07:43.658278 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:43.658240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:07:52.959365 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:07:52.959313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:08:02.260396 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:02.260301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:08:09.663203 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:09.663164    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:08:17.854517 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:17.854479    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:08:21.275675 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:21.275625    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:08:25.661828 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:25.661793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:08:31.180863 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:08:31.180830    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:15.112987 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:09:15.112953    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:26.324942 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:09:26.324848    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:32.224581 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:09:32.224546    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:32.325807 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:09:32.325773    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:09:54.456749 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:09:54.456690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:10:01.053390 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:01.053352    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:10:10.656126 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:10.656093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:10:14.597826 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:14.597731    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:18.159623 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:18.159586    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:10:20.764956 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:20.764918    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:29.758775 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:29.758730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:10:58.217696 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:58.217618    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:10:58.987544 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:10:58.987498    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:11:01.763115 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:01.763075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:11.064793 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:11.064757    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:17.759045 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.759011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:17.958271 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958237    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:17.958623 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958609    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:11:17.958668 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958628    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:11:17.958668 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:11:17.958668 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958645    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8bfa079-a16b-4b05-b52d-ffdeac8cca03" containerName="istio-proxy"
Sep 11 22:11:17.958789 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.958699    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="001fbf53-831b-4603-9fb3-2f2c3d608c53" containerName="istio-proxy"
Sep 11 22:11:17.961765 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.961734    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:17.966291 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.966264    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:11:17.966532 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.966312    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 22:11:17.966532 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.966360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:11:17.966532 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.966310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:11:17.966532 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.966310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2kttz\""
Sep 11 22:11:17.975315 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.975292    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:17.978465 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.978444    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"]
Sep 11 22:11:17.981797 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.981780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:17.994354 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:17.994324    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"]
Sep 11 22:11:18.041574 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.041574 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvsr\" (UniqueName: \"kubernetes.io/projected/70e46f1c-28d2-4137-8e09-f260b7152319-kube-api-access-4jvsr\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.041574 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e46f1c-28d2-4137-8e09-f260b7152319-openshift-service-ca\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.041910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.041910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041615    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-metrics-certs\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.041910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041645    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.041910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvq7g\" (UniqueName: \"kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.041910 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.041767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-maas-api-tls\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.143294 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.143294 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvsr\" (UniqueName: \"kubernetes.io/projected/70e46f1c-28d2-4137-8e09-f260b7152319-kube-api-access-4jvsr\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e46f1c-28d2-4137-8e09-f260b7152319-openshift-service-ca\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-metrics-certs\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvq7g\" (UniqueName: \"kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.143553 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.143472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-maas-api-tls\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.144198 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.144171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.144307 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.144204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e46f1c-28d2-4137-8e09-f260b7152319-openshift-service-ca\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.146088 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.146058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.146199 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.146170    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.146299 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.146280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-maas-api-tls\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.146299 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.146293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70e46f1c-28d2-4137-8e09-f260b7152319-metrics-certs\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.150807 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.150786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvq7g\" (UniqueName: \"kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g\") pod \"maas-api-85cff974b6-t7g4k\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") " pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.151081 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.151061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvsr\" (UniqueName: \"kubernetes.io/projected/70e46f1c-28d2-4137-8e09-f260b7152319-kube-api-access-4jvsr\") pod \"maas-api-5c564f7cc4-89gss\" (UID: \"70e46f1c-28d2-4137-8e09-f260b7152319\") " pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.272099 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.272058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:18.296406 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.296329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:18.548618 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.548591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:18.550899 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:11:18.550866    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12d9934_8f84_4689_9837_008a6f09b56d.slice/crio-6da463337a5e8345c2e1034499a7cccb9cbc6fc1fe8b773bd6332fe5b65cf9a7 WatchSource:0}: Error finding container 6da463337a5e8345c2e1034499a7cccb9cbc6fc1fe8b773bd6332fe5b65cf9a7: Status 404 returned error can't find the container with id 6da463337a5e8345c2e1034499a7cccb9cbc6fc1fe8b773bd6332fe5b65cf9a7
Sep 11 22:11:18.579428 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.579401    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"]
Sep 11 22:11:18.581909 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:11:18.581874    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e46f1c_28d2_4137_8e09_f260b7152319.slice/crio-6c452926455668432a4f87dc2cc359f02228ecd23a606e69c38b41b1f98c810d WatchSource:0}: Error finding container 6c452926455668432a4f87dc2cc359f02228ecd23a606e69c38b41b1f98c810d: Status 404 returned error can't find the container with id 6c452926455668432a4f87dc2cc359f02228ecd23a606e69c38b41b1f98c810d
Sep 11 22:11:18.950530 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.950443    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" event={"ID":"c12d9934-8f84-4689-9837-008a6f09b56d","Type":"ContainerStarted","Data":"6da463337a5e8345c2e1034499a7cccb9cbc6fc1fe8b773bd6332fe5b65cf9a7"}
Sep 11 22:11:18.951829 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:18.951800    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss" event={"ID":"70e46f1c-28d2-4137-8e09-f260b7152319","Type":"ContainerStarted","Data":"6c452926455668432a4f87dc2cc359f02228ecd23a606e69c38b41b1f98c810d"}
Sep 11 22:11:19.958443 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.958401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" event={"ID":"c12d9934-8f84-4689-9837-008a6f09b56d","Type":"ContainerStarted","Data":"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"}
Sep 11 22:11:19.958990 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.958499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:19.960007 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.959984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss" event={"ID":"70e46f1c-28d2-4137-8e09-f260b7152319","Type":"ContainerStarted","Data":"6eb3c0fc2368097bca1dbf52d5dc3ad57144872d14058069f3fd6e55ebce7018"}
Sep 11 22:11:19.960139 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.960122    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:19.973642 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.973593    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" podStartSLOduration=2.566936193 podStartE2EDuration="2.973579188s" totalImagesPullingTime="406.642995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:19.973463228 +0000 UTC m=+2445.936487877" watchObservedRunningTime="2026-09-11 22:11:19.973579188 +0000 UTC m=+2445.936603834"
Sep 11 22:11:19.988477 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:19.988425    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss" podStartSLOduration=2.60036241 podStartE2EDuration="2.988398411s" totalImagesPullingTime="388.036001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:19.987587944 +0000 UTC m=+2445.950612594" watchObservedRunningTime="2026-09-11 22:11:19.988398411 +0000 UTC m=+2445.951423073"
Sep 11 22:11:25.261125 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:25.261085    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:25.971061 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:25.971021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:25.971354 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:25.971332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c564f7cc4-89gss"
Sep 11 22:11:26.030082 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.030049    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:26.030262 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.030243    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" podUID="c12d9934-8f84-4689-9837-008a6f09b56d" containerName="maas-api" containerID="cri-o://5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21" gracePeriod=30
Sep 11 22:11:26.277827 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.277796    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:26.322840 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.322800    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls\") pod \"c12d9934-8f84-4689-9837-008a6f09b56d\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") "
Sep 11 22:11:26.323056 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.322929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs\") pod \"c12d9934-8f84-4689-9837-008a6f09b56d\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") "
Sep 11 22:11:26.323056 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.322951    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca\") pod \"c12d9934-8f84-4689-9837-008a6f09b56d\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") "
Sep 11 22:11:26.323056 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.322977    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvq7g\" (UniqueName: \"kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g\") pod \"c12d9934-8f84-4689-9837-008a6f09b56d\" (UID: \"c12d9934-8f84-4689-9837-008a6f09b56d\") "
Sep 11 22:11:26.323399 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.323368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c12d9934-8f84-4689-9837-008a6f09b56d" (UID: "c12d9934-8f84-4689-9837-008a6f09b56d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:11:26.325247 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.325212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c12d9934-8f84-4689-9837-008a6f09b56d" (UID: "c12d9934-8f84-4689-9837-008a6f09b56d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:26.325368 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.325240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c12d9934-8f84-4689-9837-008a6f09b56d" (UID: "c12d9934-8f84-4689-9837-008a6f09b56d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:26.325368 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.325308    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g" (OuterVolumeSpecName: "kube-api-access-gvq7g") pod "c12d9934-8f84-4689-9837-008a6f09b56d" (UID: "c12d9934-8f84-4689-9837-008a6f09b56d"). InnerVolumeSpecName "kube-api-access-gvq7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:26.424123 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.424084    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-metrics-certs\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:11:26.424123 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.424123    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c12d9934-8f84-4689-9837-008a6f09b56d-openshift-service-ca\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:11:26.424353 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.424136    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvq7g\" (UniqueName: \"kubernetes.io/projected/c12d9934-8f84-4689-9837-008a6f09b56d-kube-api-access-gvq7g\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:11:26.424353 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.424149    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c12d9934-8f84-4689-9837-008a6f09b56d-maas-api-tls\") on node \"ip-10-0-141-125.ec2.internal\" DevicePath \"\""
Sep 11 22:11:26.995999 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.995958    2582 generic.go:356] "Generic (PLEG): container finished" podID="c12d9934-8f84-4689-9837-008a6f09b56d" containerID="5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21" exitCode=0
Sep 11 22:11:26.996222 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.996020    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"
Sep 11 22:11:26.996222 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.996042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" event={"ID":"c12d9934-8f84-4689-9837-008a6f09b56d","Type":"ContainerDied","Data":"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"}
Sep 11 22:11:26.996222 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.996138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k" event={"ID":"c12d9934-8f84-4689-9837-008a6f09b56d","Type":"ContainerDied","Data":"6da463337a5e8345c2e1034499a7cccb9cbc6fc1fe8b773bd6332fe5b65cf9a7"}
Sep 11 22:11:26.996222 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:26.996171    2582 scope.go:117] "RemoveContainer" containerID="5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"
Sep 11 22:11:27.005332 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:27.005310    2582 scope.go:117] "RemoveContainer" containerID="5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"
Sep 11 22:11:27.005646 ip-10-0-141-125 kubenswrapper[2582]: E0911 22:11:27.005629    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21\": container with ID starting with 5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21 not found: ID does not exist" containerID="5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"
Sep 11 22:11:27.005700 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:27.005657    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21"} err="failed to get container status \"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21\": rpc error: code = NotFound desc = could not find container \"5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21\": container with ID starting with 5bab0c88d13259a9a693497f5d7383d2f3c6d772abe64bc41560f95803ad7c21 not found: ID does not exist"
Sep 11 22:11:27.011092 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:27.011046    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:27.014995 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:27.014967    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cff974b6-t7g4k"]
Sep 11 22:11:28.454768 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:28.454703    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12d9934-8f84-4689-9837-008a6f09b56d" path="/var/lib/kubelet/pods/c12d9934-8f84-4689-9837-008a6f09b56d/volumes"
Sep 11 22:11:33.058626 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:33.058590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:34.900615 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:34.900580    2582 scope.go:117] "RemoveContainer" containerID="f4446b9da1aad678481329605f196248a7a1c25a43d0d6a38f2f582fb9307e96"
Sep 11 22:11:34.909111 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:34.909089    2582 scope.go:117] "RemoveContainer" containerID="5351d722709512196ba2b291feb699e1838ce6cb8b519d66bc40116645c77494"
Sep 11 22:11:39.155257 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:39.155220    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:46.359139 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:46.359102    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bplxp"]
Sep 11 22:11:54.770984 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:54.770914    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f59564f6b-p7nlw_7a42313a-c78f-4201-b684-02867ae09065/manager/0.log"
Sep 11 22:11:54.883060 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:54.883032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dm8lc_f082e519-3d93-4903-9007-de5ae68d2066/server/0.log"
Sep 11 22:11:55.005632 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:55.005569    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pb4dd_1ba6e2e3-6730-4cba-a697-e11f9ca8638b/manager/0.log"
Sep 11 22:11:55.352968 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:55.352939    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-rl657_4073f1b0-19ff-4a45-bfa5-e817f0a14ebe/copy-manifests/0.log"
Sep 11 22:11:55.359939 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:55.359898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-rl657_4073f1b0-19ff-4a45-bfa5-e817f0a14ebe/manager/0.log"
Sep 11 22:11:55.811090 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:55.811033    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c564f7cc4-89gss_70e46f1c-28d2-4137-8e09-f260b7152319/maas-api/0.log"
Sep 11 22:11:56.027377 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:56.027344    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819400-kgn4n_d058e819-ff18-46da-b30d-7ee832fb7a34/cleanup/0.log"
Sep 11 22:11:57.514304 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:57.514268    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bplxp_94fd6c68-511d-43f8-a0ff-91531fbfc58a/limitador/0.log"
Sep 11 22:11:58.129205 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:58.129169    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbn5h_a1ff451c-9041-4516-a22c-e358c216d096/discovery/0.log"
Sep 11 22:11:58.592935 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:58.592898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8gmbg_98656f8b-93e9-4f6c-972e-9823cf5b908a/payload-pre-processing/0.log"
Sep 11 22:11:59.785822 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:11:59.785790    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ndcbh_d477bf29-b6dc-48b7-86b1-b804a0f338dc/main/0.log"
Sep 11 22:12:05.638871 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:05.638828    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:05.738511 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:05.738473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:08.497173 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:08.497136    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-shbrv_ee5ac432-0461-4ede-ac81-49fa12097464/global-pull-secret-syncer/0.log"
Sep 11 22:12:08.553631 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:08.553600    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log"
Sep 11 22:12:08.686594 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:08.686557    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-125.ec2.internal_91a6bb631a6b60831cd09f18b64e9aa6/haproxy/0.log"
Sep 11 22:12:13.860904 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:13.860877    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bplxp_94fd6c68-511d-43f8-a0ff-91531fbfc58a/limitador/0.log"
Sep 11 22:12:15.725854 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:15.725822    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hl4k_aba9dd1c-1f2d-4969-8a68-459379963079/node-exporter/0.log"
Sep 11 22:12:15.743495 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:15.743462    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hl4k_aba9dd1c-1f2d-4969-8a68-459379963079/kube-rbac-proxy/0.log"
Sep 11 22:12:15.762261 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:15.762237    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hl4k_aba9dd1c-1f2d-4969-8a68-459379963079/init-textfile/0.log"
Sep 11 22:12:16.069462 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.069434    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/prometheus/0.log"
Sep 11 22:12:16.088412 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.088384    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/config-reloader/0.log"
Sep 11 22:12:16.106589 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.106563    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/thanos-sidecar/0.log"
Sep 11 22:12:16.126431 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.126406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/kube-rbac-proxy-web/0.log"
Sep 11 22:12:16.147421 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.147401    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/kube-rbac-proxy/0.log"
Sep 11 22:12:16.170021 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.169990    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/kube-rbac-proxy-thanos/0.log"
Sep 11 22:12:16.206181 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.206157    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5db98f50-d6e5-4b54-9aad-a6f444826bfd/init-config-reloader/0.log"
Sep 11 22:12:16.236347 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.236313    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-q99jd_bbda4e59-3f20-4efa-bb14-cccfcae0dda9/prometheus-operator/0.log"
Sep 11 22:12:16.251402 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.251374    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-q99jd_bbda4e59-3f20-4efa-bb14-cccfcae0dda9/kube-rbac-proxy/0.log"
Sep 11 22:12:16.280998 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.280968    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-7jngs_558736c1-1f2d-4323-aed2-78f43ea8f769/prometheus-operator-admission-webhook/0.log"
Sep 11 22:12:16.390619 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.390588    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/thanos-query/0.log"
Sep 11 22:12:16.411916 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.411890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/kube-rbac-proxy-web/0.log"
Sep 11 22:12:16.430522 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.430494    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/kube-rbac-proxy/0.log"
Sep 11 22:12:16.450815 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.450784    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/prom-label-proxy/0.log"
Sep 11 22:12:16.469934 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.469905    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/kube-rbac-proxy-rules/0.log"
Sep 11 22:12:16.489371 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:16.489341    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65f5c67547-4wq9w_487e1406-1338-4a87-b700-d28debb5ffcc/kube-rbac-proxy-metrics/0.log"
Sep 11 22:12:18.746338 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.746299    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"]
Sep 11 22:12:18.746908 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.746888    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12d9934-8f84-4689-9837-008a6f09b56d" containerName="maas-api"
Sep 11 22:12:18.746986 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.746913    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12d9934-8f84-4689-9837-008a6f09b56d" containerName="maas-api"
Sep 11 22:12:18.747049 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.747040    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12d9934-8f84-4689-9837-008a6f09b56d" containerName="maas-api"
Sep 11 22:12:18.750047 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.750026    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.752277 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.752258    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlxlj\"/\"kube-root-ca.crt\""
Sep 11 22:12:18.752390 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.752364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wlxlj\"/\"openshift-service-ca.crt\""
Sep 11 22:12:18.752939 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.752922    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wlxlj\"/\"default-dockercfg-rq6g6\""
Sep 11 22:12:18.758793 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.758773    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"]
Sep 11 22:12:18.809173 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.809144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-sys\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.809357 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.809180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-proc\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.809357 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.809233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64h9t\" (UniqueName: \"kubernetes.io/projected/33d672d9-eac6-4561-91a4-178a9f87881a-kube-api-access-64h9t\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.809357 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.809265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-lib-modules\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.809357 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.809289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-podres\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910074 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-sys\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-proc\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910129    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64h9t\" (UniqueName: \"kubernetes.io/projected/33d672d9-eac6-4561-91a4-178a9f87881a-kube-api-access-64h9t\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-lib-modules\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-podres\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910266 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-sys\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910517 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-proc\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910517 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-lib-modules\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.910517 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.910386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/33d672d9-eac6-4561-91a4-178a9f87881a-podres\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:18.918180 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:18.918150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64h9t\" (UniqueName: \"kubernetes.io/projected/33d672d9-eac6-4561-91a4-178a9f87881a-kube-api-access-64h9t\") pod \"perf-node-gather-daemonset-bjccz\" (UID: \"33d672d9-eac6-4561-91a4-178a9f87881a\") " pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:19.061094 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:19.061015    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:19.332042 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:19.332014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"]
Sep 11 22:12:19.334466 ip-10-0-141-125 kubenswrapper[2582]: W0911 22:12:19.334442    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33d672d9_eac6_4561_91a4_178a9f87881a.slice/crio-5b1857ef43469bd67d5df551908e3976ecfc1befb0d711923cfd09ef74203b20 WatchSource:0}: Error finding container 5b1857ef43469bd67d5df551908e3976ecfc1befb0d711923cfd09ef74203b20: Status 404 returned error can't find the container with id 5b1857ef43469bd67d5df551908e3976ecfc1befb0d711923cfd09ef74203b20
Sep 11 22:12:20.246490 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:20.246454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz" event={"ID":"33d672d9-eac6-4561-91a4-178a9f87881a","Type":"ContainerStarted","Data":"2eb942fee1ec663330b8a76bdfd1329862480306a3ddde8ddaa268493f89090c"}
Sep 11 22:12:20.247009 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:20.246510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz" event={"ID":"33d672d9-eac6-4561-91a4-178a9f87881a","Type":"ContainerStarted","Data":"5b1857ef43469bd67d5df551908e3976ecfc1befb0d711923cfd09ef74203b20"}
Sep 11 22:12:20.247009 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:20.246556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:20.262569 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:20.262519    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz" podStartSLOduration=2.262505342 podStartE2EDuration="2.262505342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 22:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:20.261043308 +0000 UTC m=+2506.224067958" watchObservedRunningTime="2026-09-11 22:12:20.262505342 +0000 UTC m=+2506.225529990"
Sep 11 22:12:21.011750 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:21.011701    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r82z5_e1f27b02-e4f3-41ad-9efe-5c24fb5a904f/dns/0.log"
Sep 11 22:12:21.031849 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:21.031824    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r82z5_e1f27b02-e4f3-41ad-9efe-5c24fb5a904f/kube-rbac-proxy/0.log"
Sep 11 22:12:21.077666 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:21.077643    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4xrb_cc5971bb-454c-4913-b734-a524a351bb66/dns-node-resolver/0.log"
Sep 11 22:12:21.651786 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:21.651739    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bqqsd_ac767800-1710-4e52-a859-2b3a9047369c/node-ca/0.log"
Sep 11 22:12:23.418281 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:23.418240    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbn5h_a1ff451c-9041-4516-a22c-e358c216d096/discovery/0.log"
Sep 11 22:12:23.535986 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:23.535949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8gmbg_98656f8b-93e9-4f6c-972e-9823cf5b908a/payload-pre-processing/0.log"
Sep 11 22:12:24.244022 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:24.243983    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6pfdg_aa94ba89-8fdd-4cd4-90d8-04ab4e2c7744/konnectivity-agent/0.log" limit=1024
Sep 11 22:12:24.479078 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:24.479045    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mdgt2_5ac5b0a2-dc38-4cee-930c-0c06a85d8172/serve-healthcheck-canary/0.log"
Sep 11 22:12:24.952438 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:24.952406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wv46w_24d43615-b6a3-40ff-a376-4c095814d988/insights-operator/0.log"
Sep 11 22:12:24.952984 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:24.952961    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-wv46w_24d43615-b6a3-40ff-a376-4c095814d988/insights-operator/1.log"
Sep 11 22:12:25.087044 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:25.087011    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqb5r_5fc0e383-1643-48e2-ac95-ecb52dd72ac3/kube-rbac-proxy/0.log"
Sep 11 22:12:25.106771 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:25.106742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqb5r_5fc0e383-1643-48e2-ac95-ecb52dd72ac3/exporter/0.log"
Sep 11 22:12:25.128870 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:25.128840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dqb5r_5fc0e383-1643-48e2-ac95-ecb52dd72ac3/extractor/0.log"
Sep 11 22:12:26.261743 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:26.261654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wlxlj/perf-node-gather-daemonset-bjccz"
Sep 11 22:12:27.759671 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:27.759644    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f59564f6b-p7nlw_7a42313a-c78f-4201-b684-02867ae09065/manager/0.log"
Sep 11 22:12:27.782454 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:27.782426    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dm8lc_f082e519-3d93-4903-9007-de5ae68d2066/server/0.log"
Sep 11 22:12:27.917829 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:27.917787    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pb4dd_1ba6e2e3-6730-4cba-a697-e11f9ca8638b/manager/0.log"
Sep 11 22:12:28.123199 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:28.123110    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-rl657_4073f1b0-19ff-4a45-bfa5-e817f0a14ebe/manager/0.log"
Sep 11 22:12:28.170248 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:28.170221    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-595996979b-rl657_4073f1b0-19ff-4a45-bfa5-e817f0a14ebe/copy-manifests/0.log"
Sep 11 22:12:29.421972 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:29.421943    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74f4c6fbc8-twn9t_a2c3e4ec-a12b-4c6b-b4d2-f67f740ba2a1/manager/0.log"
Sep 11 22:12:29.448908 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:29.448855    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mnnfk_24c59373-aeb4-49d2-aade-396c566b7143/openshift-lws-operator/0.log"
Sep 11 22:12:29.671209 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:29.671175    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbn5h_a1ff451c-9041-4516-a22c-e358c216d096/discovery/0.log"
Sep 11 22:12:29.752292 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:29.752254    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8gmbg_98656f8b-93e9-4f6c-972e-9823cf5b908a/payload-pre-processing/0.log"
Sep 11 22:12:32.177627 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:32.177592    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbn5h_a1ff451c-9041-4516-a22c-e358c216d096/discovery/0.log"
Sep 11 22:12:32.283427 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:32.283398    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-8gmbg_98656f8b-93e9-4f6c-972e-9823cf5b908a/payload-pre-processing/0.log"
Sep 11 22:12:35.719100 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.719070    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6c5gv_584ee74d-2923-46fe-a902-cfffb43f7d32/kube-multus/0.log"
Sep 11 22:12:35.756106 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.756080    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/kube-multus-additional-cni-plugins/0.log"
Sep 11 22:12:35.799134 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.799082    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/egress-router-binary-copy/0.log"
Sep 11 22:12:35.832534 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.832509    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/cni-plugins/0.log"
Sep 11 22:12:35.853583 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.853554    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/bond-cni-plugin/0.log"
Sep 11 22:12:35.875424 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.875393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/routeoverride-cni/0.log"
Sep 11 22:12:35.895690 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.895668    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/whereabouts-cni-bincopy/0.log"
Sep 11 22:12:35.913956 ip-10-0-141-125 kubenswrapper[2582]: I0911 22:12:35.913934    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gnknv_b85435a6-55d3-4b79-997c-15a41ae02555/whereabouts-cni/0.log"