Sep 04 22:45:14.581336 ip-10-0-134-172 systemd[1]: Starting Kubernetes Kubelet...
Sep 04 22:45:14.982836 ip-10-0-134-172 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:45:14.982836 ip-10-0-134-172 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 04 22:45:14.982836 ip-10-0-134-172 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:45:14.982836 ip-10-0-134-172 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 04 22:45:14.982836 ip-10-0-134-172 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:45:14.983534 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.982893    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 04 22:45:14.986494 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986478    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:14.986494 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986494    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986498    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986501    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986505    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986510    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986513    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986515    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986518    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986521    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986523    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986526    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986528    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986531    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986534    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986537    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986539    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986542    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986545    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986548    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986550    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:14.986558 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986553    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986555    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986558    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986561    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986567    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986570    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986573    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986576    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986578    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986581    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986583    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986586    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986588    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986591    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986593    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986596    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986599    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986601    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986604    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:14.987203 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986606    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986609    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986614    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986617    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986620    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986623    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986626    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986628    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986630    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986633    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986635    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986638    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986640    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986643    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986647    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986650    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986653    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986655    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986658    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986661    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:14.987979 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986663    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986666    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986669    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986671    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986674    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986676    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986679    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986681    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986685    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986689    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986692    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986695    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986698    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986700    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986703    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986706    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986708    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986710    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986713    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986715    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:14.988623 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986718    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986720    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986723    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986725    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986729    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986734    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986737    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986740    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986743    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986746    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986749    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986753    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986755    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986758    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986761    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986763    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986766    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.986769    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988595    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:14.989127 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988607    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988611    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988614    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988618    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988622    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988626    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988629    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988632    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988634    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988637    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988640    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988643    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988645    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988649    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988651    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988654    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988657    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988660    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988662    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988665    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:14.989615 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988667    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988669    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988674    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988677    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988680    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988683    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988685    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988688    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988694    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988697    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988700    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988702    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988707    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988709    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988712    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988716    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988720    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988723    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988726    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:14.990136 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988729    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988731    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988734    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988737    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988739    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988742    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988744    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988747    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988750    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988752    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988756    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988758    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988761    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988764    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988767    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988769    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988772    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988775    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988777    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988781    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:14.990596 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988783    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988786    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988788    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988791    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988793    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988796    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988799    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988801    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988804    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988806    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988809    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988811    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988814    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988816    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988819    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988821    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988824    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988826    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988829    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988831    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:14.991110 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988834    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988837    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988839    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988842    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988845    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988847    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988850    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988853    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988855    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988858    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988860    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988862    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988865    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988867    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988870    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988872    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988875    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988878    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.988880    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989290    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:14.991605 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989296    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989299    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989301    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989304    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989308    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989310    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989313    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989315    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989318    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989320    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989323    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989326    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989328    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989331    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989333    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989337    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989341    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989344    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989346    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989349    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:14.992132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989352    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989354    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989357    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989359    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989362    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989364    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989367    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989370    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989372    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989375    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989378    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989380    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989383    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989388    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989391    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989394    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989398    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989400    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989403    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:14.992621 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989406    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989409    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989411    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989414    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989416    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989419    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989421    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989424    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989427    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989430    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989433    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989435    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989438    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989441    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989443    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989446    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989449    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989451    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989454    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989456    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:14.993099 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989459    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989462    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989465    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989468    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989472    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989475    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989480    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989483    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989485    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989488    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989491    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989493    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989496    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989498    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989501    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989503    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989505    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989508    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989510    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:14.993628 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989513    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989516    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989518    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989521    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989524    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989526    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989529    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989531    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989533    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989536    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989538    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989541    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989543    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989546    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989548    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989551    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989554    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989556    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989559    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.989561    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989633    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 04 22:45:14.994100 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989668    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989675    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989680    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989684    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989688    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989692    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989697    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989700    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989703    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989707    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989710    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989713    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989716    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989720    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989723    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989727    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989730    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989734    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989737    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989740    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989743    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989746    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989754    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989757    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 04 22:45:14.994598 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989760    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989763    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989767    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989770    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989773    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989776    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989779    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989783    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989786    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989789    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989796    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989799    2577 flags.go:64] FLAG: --enable-server="true"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989803    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989808    2577 flags.go:64] FLAG: --event-burst="100"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989811    2577 flags.go:64] FLAG: --event-qps="50"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989814    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989817    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989820    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989823    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989827    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989830    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989833    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989836    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989838    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989842    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 04 22:45:14.995263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989845    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989848    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989850    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989853    2577 flags.go:64] FLAG: --feature-gates=""
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989857    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989860    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989863    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989866    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989869    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989872    2577 flags.go:64] FLAG: --help="false"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989874    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989877    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989880    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989883    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989886    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989889    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989892    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989895    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989897    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989900    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989903    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989906    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989908    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989911    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 04 22:45:14.995883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989914    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989917    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989921    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989924    2577 flags.go:64] FLAG: --lock-file=""
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989927    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989930    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989933    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989938    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989941    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989944    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989947    2577 flags.go:64] FLAG: --logging-format="text"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989950    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989953    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989956    2577 flags.go:64] FLAG: --manifest-url=""
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989959    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989963    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989966    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989970    2577 flags.go:64] FLAG: --max-pods="110"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989973    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989976    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989979    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989982    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989985    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.989988    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990005    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 04 22:45:14.996476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990015    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990019    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990022    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990025    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990028    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990033    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990037    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990040    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990043    2577 flags.go:64] FLAG: --port="10250"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990047    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990050    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-029a7e0ef212405b5"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990054    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990057    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990060    2577 flags.go:64] FLAG: --register-node="true"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990062    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990066    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990070    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990073    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990075    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990081    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990084    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990087    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990090    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990093    2577 flags.go:64] FLAG: --runonce="false"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990096    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 04 22:45:14.997131 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990099    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990102    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990105    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990108    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990110    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990114    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990117    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990120    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990122    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990125    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990128    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990131    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990134    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990137    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990142    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990145    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990149    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990154    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990156    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990159    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990162    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990165    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990168    2577 flags.go:64] FLAG: --v="2"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990172    2577 flags.go:64] FLAG: --version="false"
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990177    2577 flags.go:64] FLAG: --vmodule=""
Sep 04 22:45:14.997749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990180    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.990185    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990302    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990305    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990309    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990311    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990314    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990316    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990319    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990322    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990325    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990328    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990330    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990333    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990335    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990338    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990340    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990343    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990346    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990348    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:14.998368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990351    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990353    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990356    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990361    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990364    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990366    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990371    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990374    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990377    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990379    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990382    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990385    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990389    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990391    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990394    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990396    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990399    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990402    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990404    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:14.998905 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990407    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990409    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990413    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990417    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990420    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990423    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990426    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990428    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990431    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990434    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990437    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990439    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990442    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990444    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990448    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990453    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990456    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990459    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990462    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:14.999423 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990464    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990467    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990470    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990473    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990476    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990478    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990483    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990486    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990488    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990491    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990493    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990497    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990501    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990504    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990506    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990509    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990511    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990514    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990516    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:14.999888 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990519    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990521    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990524    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990526    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990529    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990531    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990534    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990536    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990539    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990544    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990546    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990549    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990551    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990554    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990556    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990559    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990561    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990563    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990566    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990570    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:15.000366 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990573    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990576    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:14.990578    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.991315    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.998402    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:14.998417    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.000022    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.000031    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000113    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000118    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000121    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000125    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000128    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000131    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000133    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:15.000861 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000136    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000139    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000141    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000144    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000146    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000150    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000152    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000155    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000158    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000161    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000164    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000166    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000169    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000172    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000174    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000177    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000179    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000185    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000188    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000190    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:15.001260 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000193    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000195    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000199    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000202    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000204    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000207    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000209    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000212    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000215    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000217    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000220    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000223    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000225    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000228    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000230    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000233    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000235    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000238    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000241    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000243    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:15.001754 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000246    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000250    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000255    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000258    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000260    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000263    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000266    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000268    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000271    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000273    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000276    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000279    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000282    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000284    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000287    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000290    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000293    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000295    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:15.002290 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000301    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000305    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000308    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000310    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000313    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000316    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000320    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000322    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000325    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000328    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000331    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000334    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000336    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000339    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000341    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000344    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000346    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000349    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000351    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:15.002758 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000354    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000357    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000359    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000362    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000365    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000368    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000372    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000376    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000379    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000382    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000385    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000388    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000390    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000392    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.000398    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:45:15.003236 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000528    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000533    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000536    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000539    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000542    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000544    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000547    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000550    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000553    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000556    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000559    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000561    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000564    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000567    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000569    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000572    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000574    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000577    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000579    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000582    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:45:15.003611 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000584    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000587    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000590    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000593    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000596    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000599    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000601    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000604    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000606    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000609    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000611    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000614    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000616    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000619    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000621    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000624    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000626    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000629    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000631    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:45:15.004109 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000635    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000639    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000642    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000645    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000647    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000650    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000653    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000655    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000658    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000660    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000663    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000665    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000668    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000670    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000673    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000675    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000678    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000681    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000684    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000687    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:45:15.004593 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000689    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000691    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000694    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000696    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000699    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000701    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000704    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000706    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000708    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000711    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000713    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000716    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000719    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000721    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000723    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000726    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000729    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000731    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000734    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000737    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:45:15.005215 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000741    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000744    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000746    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000749    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000752    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000754    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000757    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000760    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000762    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000765    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000767    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000770    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000772    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000775    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000779    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000782    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000785    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000788    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000791    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:45:15.005709 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:15.000793    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.000799    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.000908    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.003104    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.004105    2577 server.go:1019] "Starting client certificate rotation"
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.004206    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 22:45:15.006193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.004246    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 22:45:15.007113 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.007093    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 22:45:15.008466 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.008447    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 22:45:15.021657 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.021627    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 04 22:45:15.024696 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.024679    2577 log.go:25] "Validated CRI v1 image API"
Sep 04 22:45:15.025471 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.025452    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 04 22:45:15.028981 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.028960    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b0ad4e55-1990-49ad-8c89-f08702c8224f:/dev/nvme0n1p4 ed6e6151-352d-463d-9205-579f372a68ef:/dev/nvme0n1p3]
Sep 04 22:45:15.029060 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.028980    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 04 22:45:15.029874 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.029856    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 22:45:15.035270 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.035150    2577 manager.go:216] Machine: {Timestamp:2026-09-04 22:45:15.033807528 +0000 UTC m=+0.333625072 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099727 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2122aedbd0f94a7d6418c725b135fe SystemUUID:ec2122ae-dbd0-f94a-7d64-18c725b135fe BootID:72b17ace-73b5-40eb-8862-500794bb969a Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:0c:c8:76:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:0c:c8:76:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:1f:78:32:89:0c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 04 22:45:15.035270 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.035259    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 04 22:45:15.035428 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.035343    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 04 22:45:15.038579 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.038551    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 04 22:45:15.038749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.038581    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-172.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 04 22:45:15.040011 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.039986    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 04 22:45:15.040011 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.040012    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 04 22:45:15.040101 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.040021    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 22:45:15.040831 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.040819    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 22:45:15.041601 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.041589    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 04 22:45:15.044607 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044594    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 22:45:15.044726 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044716    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 04 22:45:15.044773 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044745    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 04 22:45:15.044773 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044756    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 04 22:45:15.044773 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044766    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 04 22:45:15.044889 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044777    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 04 22:45:15.044889 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.044790    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 04 22:45:15.045253 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.045239    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 22:45:15.045314 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.045272    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 22:45:15.047583 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.047561    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 04 22:45:15.050047 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.050025    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 04 22:45:15.050100 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.050063    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 04 22:45:15.051024 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051002    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051026    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051037    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051043    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051052    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051058    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 04 22:45:15.051070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051071    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 04 22:45:15.051244 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051077    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 04 22:45:15.051244 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051093    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 04 22:45:15.051244 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051099    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 04 22:45:15.051244 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051107    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 04 22:45:15.051244 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051112    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 04 22:45:15.051697 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051687    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 04 22:45:15.051732 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.051699    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 04 22:45:15.053449 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.053430    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-172.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 04 22:45:15.053834 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.053807    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 04 22:45:15.053906 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.053807    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-172.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 04 22:45:15.055694 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.055675    2577 server.go:1268] "Started kubelet"
Sep 04 22:45:15.055765 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.055723    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 04 22:45:15.055857 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.055799    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 04 22:45:15.055911 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.055880    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 04 22:45:15.056284 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.056266    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7jd2k"
Sep 04 22:45:15.056641 ip-10-0-134-172 systemd[1]: Started Kubernetes Kubelet.
Sep 04 22:45:15.059478 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.059409    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 04 22:45:15.060928 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.060909    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 04 22:45:15.061371 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.061348    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 04 22:45:15.063823 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.063803    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 04 22:45:15.063823 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.063816    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 04 22:45:15.063988 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.063967    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:15.064296 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064268    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7jd2k"
Sep 04 22:45:15.064484 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064466    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 04 22:45:15.064548 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064488    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 04 22:45:15.064548 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064469    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 04 22:45:15.064635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064602    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 04 22:45:15.064635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.064611    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 04 22:45:15.065095 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065078    2577 factory.go:55] Registering systemd factory
Sep 04 22:45:15.065166 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065098    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 04 22:45:15.065382 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065366    2577 factory.go:153] Registering CRI-O factory
Sep 04 22:45:15.065382 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065383    2577 factory.go:223] Registration of the crio container factory successfully
Sep 04 22:45:15.065508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065430    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 04 22:45:15.065508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065448    2577 factory.go:103] Registering Raw factory
Sep 04 22:45:15.065508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065461    2577 manager.go:1196] Started watching for new ooms in manager
Sep 04 22:45:15.065919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.065897    2577 manager.go:349] Starting recovery of all containers
Sep 04 22:45:15.066145 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.065022    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-172.ec2.internal.18d23f95e5adc92e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-172.ec2.internal,UID:ip-10-0-134-172.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-172.ec2.internal,},FirstTimestamp:2026-09-04 22:45:15.05555691 +0000 UTC m=+0.355374455,LastTimestamp:2026-09-04 22:45:15.05555691 +0000 UTC m=+0.355374455,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-172.ec2.internal,}"
Sep 04 22:45:15.074342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.074309    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:15.077350 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.077197    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-172.ec2.internal\" not found" node="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.083180 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.083165    2577 manager.go:354] Recovery completed
Sep 04 22:45:15.088284 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.088270    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.090071 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090056    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.090158 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090086    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.090158 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090100    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.090615 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090593    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 04 22:45:15.090615 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090607    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 04 22:45:15.090731 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090632    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 22:45:15.090731 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.090649    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.092022 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.091985    2577 policy_none.go:49] "None policy: Start"
Sep 04 22:45:15.092083 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.092028    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 04 22:45:15.092083 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.092041    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 04 22:45:15.092579 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.092562    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.092642 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.092591    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.092642 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.092602    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.093080 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.093068    2577 policy_none.go:47] "Start"
Sep 04 22:45:15.127038 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.126639    2577 manager.go:337] "Starting Device Plugin manager"
Sep 04 22:45:15.127038 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.126700    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 04 22:45:15.127038 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.126715    2577 server.go:87] "Starting device plugin registration server"
Sep 04 22:45:15.127201 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.127183    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 04 22:45:15.127236 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.127196    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 04 22:45:15.127316 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.127299    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 04 22:45:15.127404 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.127389    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 04 22:45:15.127452 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.127407    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 04 22:45:15.127940 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.127918    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 04 22:45:15.128035 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.127966    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:15.189072 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.189025    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 04 22:45:15.190221 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.190199    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 04 22:45:15.190319 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.190271    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 04 22:45:15.190319 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.190303    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 04 22:45:15.190431 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.190414    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 04 22:45:15.192848 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.192830    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:15.227943 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.227914    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.228877 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.228859    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.228972 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.228895    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.228972 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.228911    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.228972 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.228941    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.236617 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.236542    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.236617 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.236570    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-172.ec2.internal\": node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:15.253490 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.253460    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:15.290499 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.290465    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"]
Sep 04 22:45:15.290574 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.290563    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.292058 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.292040    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.292145 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.292073    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.292145 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.292084    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.293347 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.293335    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.293460 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.293445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.293505 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.293474    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.294070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294051    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.294143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294080    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.294143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294095    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.294143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294098    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.294143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294115    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.294143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.294127    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.295144 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.295130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.295186 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.295157    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:45:15.295810 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.295795    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:45:15.295874 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.295825    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:45:15.295874 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.295840    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:45:15.332839 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.332811    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-172.ec2.internal\" not found" node="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.337102 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.337085    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-172.ec2.internal\" not found" node="ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.354503 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.354471    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:15.366431 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.366411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90d483920ce499f7a5df446cbd8da0f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-172.ec2.internal\" (UID: \"e90d483920ce499f7a5df446cbd8da0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.366543 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.366441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.366543 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:15.366469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:15.455076 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:15.455043    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:16.596253 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90d483920ce499f7a5df446cbd8da0f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-172.ec2.internal\" (UID: \"e90d483920ce499f7a5df446cbd8da0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.596323 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90d483920ce499f7a5df446cbd8da0f-config\") pod \"kube-apiserver-proxy-ip-10-0-134-172.ec2.internal\" (UID: \"e90d483920ce499f7a5df446cbd8da0f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.596369 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.596369 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.596440 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.596440 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.596403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7465875ab6954932567f18fa44a4de7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal\" (UID: \"e7465875ab6954932567f18fa44a4de7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.684681 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:16.684609    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:16.764133 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.764097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.769137 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:16.769117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:16.785533 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:16.785498    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:16.886039 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:16.885993    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:16.986628 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:16.986543    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:17.023633 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.023606    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:17.086901 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.086864    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-172.ec2.internal\" not found"
Sep 04 22:45:17.132424 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.132391    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 04 22:45:17.132900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.132548    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 22:45:17.132900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.132585    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 22:45:17.132900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.132585    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 22:45:17.146480 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.146451    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:17.174275 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.174248    2577 apiserver.go:52] "Watching apiserver"
Sep 04 22:45:17.181847 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.181821    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 04 22:45:17.183251 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.183228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sxmgs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv","openshift-dns/node-resolver-nhj5d","openshift-multus/multus-qhmhp","openshift-multus/network-metrics-daemon-mcpvb","openshift-network-diagnostics/network-check-target-nh76j","openshift-network-operator/iptables-alerter-kgdwx","kube-system/global-pull-secret-syncer-c75hg","openshift-cluster-node-tuning-operator/tuned-bxqcj","openshift-image-registry/node-ca-m424m","openshift-multus/multus-additional-cni-plugins-m4fgs","openshift-ovn-kubernetes/ovnkube-node-p66rd"]
Sep 04 22:45:17.185096 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.185074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.186270 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.186251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.187387 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.187366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.187756 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.187530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jfvms\""
Sep 04 22:45:17.187756 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.187536    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 04 22:45:17.187929 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.187826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 04 22:45:17.188266 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.188250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 04 22:45:17.188431 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.188417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.188540 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.188524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.188946 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.188919    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.189389 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.189365    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cbbz8\""
Sep 04 22:45:17.190414 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.190373    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.191260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.190679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-55q5t\""
Sep 04 22:45:17.191260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.190844    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.191260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.190911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.191260 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.190988    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:17.191554 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.191525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 04 22:45:17.191830 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.191811    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.191931 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.191903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7p6jb\""
Sep 04 22:45:17.192000 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.191985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.192043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.192008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 04 22:45:17.192380 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.192363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:17.192554 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.192475    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:17.192862 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.192656    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 04 22:45:17.192862 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.192678    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:17.194905 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.194888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.196035 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.196021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.196097 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.196075    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:17.197206 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.197951 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197905    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-03 22:40:16 +0000 UTC" deadline="2028-03-11 06:33:40.525704348 +0000 UTC"
Sep 04 22:45:17.197951 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197949    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13279h48m23.327759062s"
Sep 04 22:45:17.198099 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.198099 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197993    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.198099 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197964    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nzd7j\""
Sep 04 22:45:17.198099 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.197966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 04 22:45:17.199389 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.199446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.199446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-cnibin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19fe17dd-bddb-4c93-ae78-72d7a45694b8-host-slash\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199483    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff9v\" (UniqueName: \"kubernetes.io/projected/19fe17dd-bddb-4c93-ae78-72d7a45694b8-kube-api-access-vff9v\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-device-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-system-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-cni-binary-copy\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4wm5\" (UniqueName: \"kubernetes.io/projected/0333a425-3e65-479b-a1b6-2419b0d01dbe-kube-api-access-k4wm5\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-socket-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-bin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sls5h\" (UniqueName: \"kubernetes.io/projected/7ca99630-4b48-417c-aff0-efe8419fdb18-kube-api-access-sls5h\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cef06c4-9dc5-4510-96de-855b61f090fb-konnectivity-ca\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de5fdc91-819b-4568-bb01-54547f4735cd-hosts-file\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5fdc91-819b-4568-bb01-54547f4735cd-tmp-dir\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-k8s-cni-cncf-io\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-conf-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19fe17dd-bddb-4c93-ae78-72d7a45694b8-iptables-alerter-script\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.199925 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-kubelet\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-registration-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.199997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtl9k\" (UniqueName: \"kubernetes.io/projected/3169f0f2-0259-4375-a406-b14f54840ff2-kube-api-access-rtl9k\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554xg\" (UniqueName: \"kubernetes.io/projected/de5fdc91-819b-4568-bb01-54547f4735cd-kube-api-access-554xg\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-multus\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-etc-kubernetes\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cef06c4-9dc5-4510-96de-855b61f090fb-agent-certs\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-hostroot\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-daemon-config\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-sys-fs\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-socket-dir-parent\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-netns\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-multus-certs\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200306    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.200446 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200309    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6qv85\""
Sep 04 22:45:17.201193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.200625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.201600 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.201582    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.201696 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.201626    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 04 22:45:17.201696 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.201647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tcjgq\""
Sep 04 22:45:17.202227 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.202206    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.202651 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.202635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.202890 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.202870    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 04 22:45:17.203010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.202877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4xsh\""
Sep 04 22:45:17.203010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.202956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 04 22:45:17.204360 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.204345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 04 22:45:17.204829 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.204813    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 04 22:45:17.205127 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205115    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 04 22:45:17.205260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205247    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 04 22:45:17.205355 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205339    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j6qff\""
Sep 04 22:45:17.205415 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 04 22:45:17.205490 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205461    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 04 22:45:17.205981 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.205963    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 22:45:17.206082 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.206049    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"
Sep 04 22:45:17.206143 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.206112    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal"]
Sep 04 22:45:17.207782 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.207726    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 22:45:17.212335 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.212315    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 22:45:17.212525 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.212508    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal"]
Sep 04 22:45:17.224852 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.224827    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sr5v7"
Sep 04 22:45:17.230605 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.230581    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sr5v7"
Sep 04 22:45:17.294707 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.294680    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 04 22:45:17.300520 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cef06c4-9dc5-4510-96de-855b61f090fb-konnectivity-ca\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.300636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5fdc91-819b-4568-bb01-54547f4735cd-tmp-dir\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.300636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-k8s-cni-cncf-io\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.300636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-kubelet\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.300636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-env-overrides\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.300636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7cz\" (UniqueName: \"kubernetes.io/projected/21916f8f-fde8-475e-bbfc-c443fcccbe80-kube-api-access-4c7cz\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-k8s-cni-cncf-io\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-tmp\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19fe17dd-bddb-4c93-ae78-72d7a45694b8-iptables-alerter-script\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-kubelet\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-etc-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-dbus\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554xg\" (UniqueName: \"kubernetes.io/projected/de5fdc91-819b-4568-bb01-54547f4735cd-kube-api-access-554xg\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.300873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-multus\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-kubelet\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de5fdc91-819b-4568-bb01-54547f4735cd-tmp-dir\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-etc-kubernetes\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-etc-kubernetes\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-systemd-units\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300928    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-multus\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-bin\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.300983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovn-node-metrics-cert\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-modprobe-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-run\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cef06c4-9dc5-4510-96de-855b61f090fb-agent-certs\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-hostroot\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7cef06c4-9dc5-4510-96de-855b61f090fb-konnectivity-ca\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-system-cni-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-hostroot\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.301289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-netd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-lib-modules\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-sys-fs\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-netns\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8llx\" (UniqueName: \"kubernetes.io/projected/959228e2-e29a-422b-af73-02a7fa386332-kube-api-access-s8llx\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-var-lib-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-node-log\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-sys-fs\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-netns\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-kubelet-config\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-var-lib-kubelet\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19fe17dd-bddb-4c93-ae78-72d7a45694b8-iptables-alerter-script\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-cnibin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301456    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-cnibin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19fe17dd-bddb-4c93-ae78-72d7a45694b8-host-slash\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.302067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19fe17dd-bddb-4c93-ae78-72d7a45694b8-host-slash\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-device-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-system-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4wm5\" (UniqueName: \"kubernetes.io/projected/0333a425-3e65-479b-a1b6-2419b0d01dbe-kube-api-access-k4wm5\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-device-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-system-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-cni-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-config\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-tuned\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-systemd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de5fdc91-819b-4568-bb01-54547f4735cd-hosts-file\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-conf-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/de5fdc91-819b-4568-bb01-54547f4735cd-hosts-file\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-conf-dir\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-log-socket\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.302728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-host\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.301999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvgs\" (UniqueName: \"kubernetes.io/projected/4ee6f1db-453b-4985-aa29-774fd08ba876-kube-api-access-jfvgs\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-registration-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtl9k\" (UniqueName: \"kubernetes.io/projected/3169f0f2-0259-4375-a406-b14f54840ff2-kube-api-access-rtl9k\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-daemon-config\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ee6f1db-453b-4985-aa29-774fd08ba876-host\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-registration-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-netns\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-script-lib\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-kubernetes\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-socket-dir-parent\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.303420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-multus-certs\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-run-multus-certs\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ee6f1db-453b-4985-aa29-774fd08ba876-serviceca\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-socket-dir-parent\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-slash\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwqs\" (UniqueName: \"kubernetes.io/projected/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-kube-api-access-vrwqs\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-ovn\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysconfig\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.302783    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-sys\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.302873    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:17.802839141 +0000 UTC m=+1.973913743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:17.303919 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff9v\" (UniqueName: \"kubernetes.io/projected/19fe17dd-bddb-4c93-ae78-72d7a45694b8-kube-api-access-vff9v\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-cni-binary-copy\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.302986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-cnibin\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-conf\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-systemd\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-socket-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-bin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sls5h\" (UniqueName: \"kubernetes.io/projected/7ca99630-4b48-417c-aff0-efe8419fdb18-kube-api-access-sls5h\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ca99630-4b48-417c-aff0-efe8419fdb18-host-var-lib-cni-bin\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-multus-daemon-config\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3169f0f2-0259-4375-a406-b14f54840ff2-socket-dir\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.304407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.303471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ca99630-4b48-417c-aff0-efe8419fdb18-cni-binary-copy\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.305073 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.305053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7cef06c4-9dc5-4510-96de-855b61f090fb-agent-certs\") pod \"konnectivity-agent-sxmgs\" (UID: \"7cef06c4-9dc5-4510-96de-855b61f090fb\") " pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.307028 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.307008    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:17.307028 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.307027    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:17.307168 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.307037    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:17.307168 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.307081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:17.807067909 +0000 UTC m=+1.978142520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:17.308808 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.308787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-554xg\" (UniqueName: \"kubernetes.io/projected/de5fdc91-819b-4568-bb01-54547f4735cd-kube-api-access-554xg\") pod \"node-resolver-nhj5d\" (UID: \"de5fdc91-819b-4568-bb01-54547f4735cd\") " pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.309386 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.309368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4wm5\" (UniqueName: \"kubernetes.io/projected/0333a425-3e65-479b-a1b6-2419b0d01dbe-kube-api-access-k4wm5\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.313933 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.313907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtl9k\" (UniqueName: \"kubernetes.io/projected/3169f0f2-0259-4375-a406-b14f54840ff2-kube-api-access-rtl9k\") pod \"aws-ebs-csi-driver-node-zwpqv\" (UID: \"3169f0f2-0259-4375-a406-b14f54840ff2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.314128 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.314110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff9v\" (UniqueName: \"kubernetes.io/projected/19fe17dd-bddb-4c93-ae78-72d7a45694b8-kube-api-access-vff9v\") pod \"iptables-alerter-kgdwx\" (UID: \"19fe17dd-bddb-4c93-ae78-72d7a45694b8\") " pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.315068 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.315050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sls5h\" (UniqueName: \"kubernetes.io/projected/7ca99630-4b48-417c-aff0-efe8419fdb18-kube-api-access-sls5h\") pod \"multus-qhmhp\" (UID: \"7ca99630-4b48-417c-aff0-efe8419fdb18\") " pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.344706 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.344680    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90d483920ce499f7a5df446cbd8da0f.slice/crio-14a985c9717215244be21b6bf7eebaeeb8bf9832bba469244104835d3a3556cb WatchSource:0}: Error finding container 14a985c9717215244be21b6bf7eebaeeb8bf9832bba469244104835d3a3556cb: Status 404 returned error can't find the container with id 14a985c9717215244be21b6bf7eebaeeb8bf9832bba469244104835d3a3556cb
Sep 04 22:45:17.345076 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.345064    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7465875ab6954932567f18fa44a4de7.slice/crio-19c927bbcd90a008e080bcd48dba03fcd4af8bc8dcf97ac7928dbbd677ff0b8d WatchSource:0}: Error finding container 19c927bbcd90a008e080bcd48dba03fcd4af8bc8dcf97ac7928dbbd677ff0b8d: Status 404 returned error can't find the container with id 19c927bbcd90a008e080bcd48dba03fcd4af8bc8dcf97ac7928dbbd677ff0b8d
Sep 04 22:45:17.351562 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.351544    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:45:17.403420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-node-log\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403420 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-kubelet-config\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-var-lib-kubelet\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-config\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-tuned\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-systemd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-node-log\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-kubelet-config\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-var-lib-kubelet\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-log-socket\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-systemd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-host\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvgs\" (UniqueName: \"kubernetes.io/projected/4ee6f1db-453b-4985-aa29-774fd08ba876-kube-api-access-jfvgs\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-log-socket\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.403629 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-host\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ee6f1db-453b-4985-aa29-774fd08ba876-host\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-netns\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ee6f1db-453b-4985-aa29-774fd08ba876-host\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-script-lib\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-netns\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.403783    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-kubernetes\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ee6f1db-453b-4985-aa29-774fd08ba876-serviceca\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.403884    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:17.903866588 +0000 UTC m=+2.074941197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-kubernetes\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-slash\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.404267 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.403980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwqs\" (UniqueName: \"kubernetes.io/projected/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-kube-api-access-vrwqs\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-ovn\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysconfig\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-sys\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-cnibin\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-config\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-conf\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-systemd\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysconfig\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-kubelet\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-cnibin\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-env-overrides\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-sys\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.405063 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-kubelet\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-conf\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-slash\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7cz\" (UniqueName: \"kubernetes.io/projected/21916f8f-fde8-475e-bbfc-c443fcccbe80-kube-api-access-4c7cz\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovnkube-script-lib\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-systemd\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-tmp\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-etc-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-dbus\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-sysctl-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/959228e2-e29a-422b-af73-02a7fa386332-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-systemd-units\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-etc-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-systemd-units\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-bin\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovn-node-metrics-cert\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1fcc2d0-572d-4bc6-91bc-44a34539f922-dbus\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21916f8f-fde8-475e-bbfc-c443fcccbe80-env-overrides\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-bin\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-run-ovn\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-modprobe-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-run\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-system-cni-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-modprobe-d\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ee6f1db-453b-4985-aa29-774fd08ba876-serviceca\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404846    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-run\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-system-cni-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-run-ovn-kubernetes\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-netd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-lib-modules\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.406840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-host-cni-netd\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8llx\" (UniqueName: \"kubernetes.io/projected/959228e2-e29a-422b-af73-02a7fa386332-kube-api-access-s8llx\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.404960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-var-lib-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.405001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/959228e2-e29a-422b-af73-02a7fa386332-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.405019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21916f8f-fde8-475e-bbfc-c443fcccbe80-var-lib-openvswitch\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.405054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-lib-modules\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.406356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-etc-tuned\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.406638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21916f8f-fde8-475e-bbfc-c443fcccbe80-ovn-node-metrics-cert\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.407300 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.406907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-tmp\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.415099 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.415070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvgs\" (UniqueName: \"kubernetes.io/projected/4ee6f1db-453b-4985-aa29-774fd08ba876-kube-api-access-jfvgs\") pod \"node-ca-m424m\" (UID: \"4ee6f1db-453b-4985-aa29-774fd08ba876\") " pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.415196 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.415123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwqs\" (UniqueName: \"kubernetes.io/projected/4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d-kube-api-access-vrwqs\") pod \"tuned-bxqcj\" (UID: \"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d\") " pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.416133 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.416112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8llx\" (UniqueName: \"kubernetes.io/projected/959228e2-e29a-422b-af73-02a7fa386332-kube-api-access-s8llx\") pod \"multus-additional-cni-plugins-m4fgs\" (UID: \"959228e2-e29a-422b-af73-02a7fa386332\") " pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.416213 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.416137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7cz\" (UniqueName: \"kubernetes.io/projected/21916f8f-fde8-475e-bbfc-c443fcccbe80-kube-api-access-4c7cz\") pod \"ovnkube-node-p66rd\" (UID: \"21916f8f-fde8-475e-bbfc-c443fcccbe80\") " pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.514616 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.514523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:17.522899 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.522872    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cef06c4_9dc5_4510_96de_855b61f090fb.slice/crio-c48788939f802f0912d11f800e01353d0281a5b4e1da7d73345a42c450436426 WatchSource:0}: Error finding container c48788939f802f0912d11f800e01353d0281a5b4e1da7d73345a42c450436426: Status 404 returned error can't find the container with id c48788939f802f0912d11f800e01353d0281a5b4e1da7d73345a42c450436426
Sep 04 22:45:17.530169 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.530148    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv"
Sep 04 22:45:17.537132 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.537107    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3169f0f2_0259_4375_a406_b14f54840ff2.slice/crio-255fc0b4bdc16f0ca6f43facc865a60fc173942c48fb1f0b50e45049b2468218 WatchSource:0}: Error finding container 255fc0b4bdc16f0ca6f43facc865a60fc173942c48fb1f0b50e45049b2468218: Status 404 returned error can't find the container with id 255fc0b4bdc16f0ca6f43facc865a60fc173942c48fb1f0b50e45049b2468218
Sep 04 22:45:17.546139 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.546107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nhj5d"
Sep 04 22:45:17.552662 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.552640    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5fdc91_819b_4568_bb01_54547f4735cd.slice/crio-4a9310f11a9de39dfc3fdd2820c750aef6f19d862f8c3112e5790db426663872 WatchSource:0}: Error finding container 4a9310f11a9de39dfc3fdd2820c750aef6f19d862f8c3112e5790db426663872: Status 404 returned error can't find the container with id 4a9310f11a9de39dfc3fdd2820c750aef6f19d862f8c3112e5790db426663872
Sep 04 22:45:17.560443 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.560424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qhmhp"
Sep 04 22:45:17.567331 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.567276    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca99630_4b48_417c_aff0_efe8419fdb18.slice/crio-6ff7691c97cd3bf84e9a6ce09fdf1c8a4669cf8a6ae2530db03cb7f24e4eafb9 WatchSource:0}: Error finding container 6ff7691c97cd3bf84e9a6ce09fdf1c8a4669cf8a6ae2530db03cb7f24e4eafb9: Status 404 returned error can't find the container with id 6ff7691c97cd3bf84e9a6ce09fdf1c8a4669cf8a6ae2530db03cb7f24e4eafb9
Sep 04 22:45:17.580449 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.580424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kgdwx"
Sep 04 22:45:17.587378 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.587356    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fe17dd_bddb_4c93_ae78_72d7a45694b8.slice/crio-92646b7cd033f39c75222e7ebd53d6e0cf40e10deddf5cc467693c03eedcf2b0 WatchSource:0}: Error finding container 92646b7cd033f39c75222e7ebd53d6e0cf40e10deddf5cc467693c03eedcf2b0: Status 404 returned error can't find the container with id 92646b7cd033f39c75222e7ebd53d6e0cf40e10deddf5cc467693c03eedcf2b0
Sep 04 22:45:17.600910 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.600887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxqcj"
Sep 04 22:45:17.608289 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.608265    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb30bf8_0f7f_4a15_9d09_3db3ee1b436d.slice/crio-963bae2a0e235ec5ec0b469374c3f4bf91291219d210d8f138bcfd4bc6a48528 WatchSource:0}: Error finding container 963bae2a0e235ec5ec0b469374c3f4bf91291219d210d8f138bcfd4bc6a48528: Status 404 returned error can't find the container with id 963bae2a0e235ec5ec0b469374c3f4bf91291219d210d8f138bcfd4bc6a48528
Sep 04 22:45:17.621186 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.621163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m424m"
Sep 04 22:45:17.628968 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.628941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee6f1db_453b_4985_aa29_774fd08ba876.slice/crio-1ba9ffd87a150cf35ad50196fad78ce1d51e046b1314ccd7f965019ee311a6ef WatchSource:0}: Error finding container 1ba9ffd87a150cf35ad50196fad78ce1d51e046b1314ccd7f965019ee311a6ef: Status 404 returned error can't find the container with id 1ba9ffd87a150cf35ad50196fad78ce1d51e046b1314ccd7f965019ee311a6ef
Sep 04 22:45:17.637250 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.637234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4fgs"
Sep 04 22:45:17.641029 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.641012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:17.645052 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.645023    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959228e2_e29a_422b_af73_02a7fa386332.slice/crio-93842ecd7c7d5bcb972fe5de7538e89c56fe431425145d5716c8ee03a0bb57f7 WatchSource:0}: Error finding container 93842ecd7c7d5bcb972fe5de7538e89c56fe431425145d5716c8ee03a0bb57f7: Status 404 returned error can't find the container with id 93842ecd7c7d5bcb972fe5de7538e89c56fe431425145d5716c8ee03a0bb57f7
Sep 04 22:45:17.649142 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:17.649115    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21916f8f_fde8_475e_bbfc_c443fcccbe80.slice/crio-f1b3c7a9f6050b5ce733b924a6d274a7552f53001705c22473994f7a45836aca WatchSource:0}: Error finding container f1b3c7a9f6050b5ce733b924a6d274a7552f53001705c22473994f7a45836aca: Status 404 returned error can't find the container with id f1b3c7a9f6050b5ce733b924a6d274a7552f53001705c22473994f7a45836aca
Sep 04 22:45:17.807328 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.807247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:17.807328 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.807322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:17.807549 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807499    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:17.807549 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807530    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:17.807549 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807549    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:17.807687 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807561    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:18.807546035 +0000 UTC m=+2.978620625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:17.807687 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807562    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:17.807687 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.807608    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:18.807597245 +0000 UTC m=+2.978671837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:17.909013 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:17.908567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:17.909013 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.908700    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:17.909013 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:17.908775    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:18.908755922 +0000 UTC m=+3.079830516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:18.233782 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.233659    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:40:17 +0000 UTC" deadline="2028-05-29 13:05:15.414789552 +0000 UTC"
Sep 04 22:45:18.233782 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.233725    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15182h19m57.181069323s"
Sep 04 22:45:18.322880 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.321994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:18.322880 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.322202    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:18.339511 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.339452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"f1b3c7a9f6050b5ce733b924a6d274a7552f53001705c22473994f7a45836aca"}
Sep 04 22:45:18.356854 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.356820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m424m" event={"ID":"4ee6f1db-453b-4985-aa29-774fd08ba876","Type":"ContainerStarted","Data":"1ba9ffd87a150cf35ad50196fad78ce1d51e046b1314ccd7f965019ee311a6ef"}
Sep 04 22:45:18.391427 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.391336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhj5d" event={"ID":"de5fdc91-819b-4568-bb01-54547f4735cd","Type":"ContainerStarted","Data":"4a9310f11a9de39dfc3fdd2820c750aef6f19d862f8c3112e5790db426663872"}
Sep 04 22:45:18.411330 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.411290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxmgs" event={"ID":"7cef06c4-9dc5-4510-96de-855b61f090fb","Type":"ContainerStarted","Data":"c48788939f802f0912d11f800e01353d0281a5b4e1da7d73345a42c450436426"}
Sep 04 22:45:18.431613 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.431559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal" event={"ID":"e90d483920ce499f7a5df446cbd8da0f","Type":"ContainerStarted","Data":"14a985c9717215244be21b6bf7eebaeeb8bf9832bba469244104835d3a3556cb"}
Sep 04 22:45:18.452055 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.452014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerStarted","Data":"93842ecd7c7d5bcb972fe5de7538e89c56fe431425145d5716c8ee03a0bb57f7"}
Sep 04 22:45:18.462249 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.462219    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:18.463897 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.463860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxqcj" event={"ID":"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d","Type":"ContainerStarted","Data":"963bae2a0e235ec5ec0b469374c3f4bf91291219d210d8f138bcfd4bc6a48528"}
Sep 04 22:45:18.469748 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.468836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kgdwx" event={"ID":"19fe17dd-bddb-4c93-ae78-72d7a45694b8","Type":"ContainerStarted","Data":"92646b7cd033f39c75222e7ebd53d6e0cf40e10deddf5cc467693c03eedcf2b0"}
Sep 04 22:45:18.484491 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.484380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhmhp" event={"ID":"7ca99630-4b48-417c-aff0-efe8419fdb18","Type":"ContainerStarted","Data":"6ff7691c97cd3bf84e9a6ce09fdf1c8a4669cf8a6ae2530db03cb7f24e4eafb9"}
Sep 04 22:45:18.490552 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.490514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv" event={"ID":"3169f0f2-0259-4375-a406-b14f54840ff2","Type":"ContainerStarted","Data":"255fc0b4bdc16f0ca6f43facc865a60fc173942c48fb1f0b50e45049b2468218"}
Sep 04 22:45:18.507718 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.507678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal" event={"ID":"e7465875ab6954932567f18fa44a4de7","Type":"ContainerStarted","Data":"19c927bbcd90a008e080bcd48dba03fcd4af8bc8dcf97ac7928dbbd677ff0b8d"}
Sep 04 22:45:18.509194 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.509170    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:18.818760 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.818660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:18.818760 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.818726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:18.818983 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.818890    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:18.818983 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.818962    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:20.818943904 +0000 UTC m=+4.990018497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:18.819420 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.819387    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:18.819420 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.819411    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:18.819545 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.819423    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:18.819545 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.819468    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:20.819454607 +0000 UTC m=+4.990529200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:18.919429 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:18.919383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:18.919611 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.919555    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:18.919675 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:18.919617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:20.919598412 +0000 UTC m=+5.090673010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:19.234365 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:19.234277    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:40:17 +0000 UTC" deadline="2028-02-02 02:16:09.196206319 +0000 UTC"
Sep 04 22:45:19.234365 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:19.234315    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12363h30m49.961896012s"
Sep 04 22:45:19.273256 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:19.273211    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:45:19.319987 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:19.319952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:19.320170 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:19.320005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:19.320170 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:19.320093    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:19.320280 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:19.320222    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:20.326486 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:20.325995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:20.326486 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.326137    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:20.836956 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:20.836918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:20.837136 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:20.837003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:20.837196 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837168    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:20.837196 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837192    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:20.837296 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837206    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:20.837296 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:24.837245667 +0000 UTC m=+9.008320261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:20.837763 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837659    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:20.837763 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.837710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:24.83769558 +0000 UTC m=+9.008770173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:20.938575 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:20.938003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:20.938575 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.938202    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:20.938575 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:20.938261    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:24.938243691 +0000 UTC m=+9.109318284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:21.320656 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:21.319992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:21.320656 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:21.320031    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:21.320656 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:21.320139    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:21.320656 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:21.320566    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:22.320290 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:22.320256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:22.320794 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:22.320409    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:23.319962 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:23.319916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:23.320156 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:23.320056    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:23.320156 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:23.320116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:23.320279 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:23.320199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:24.319770 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:24.319718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:24.320244 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.319891    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:24.872466 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:24.872398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:24.872650 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:24.872475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:24.872650 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872621    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:24.872804 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872657    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:24.872804 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872681    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:32.872665853 +0000 UTC m=+17.043740446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:24.872804 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872710    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:24.872804 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872725    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:24.872804 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.872799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:32.872781726 +0000 UTC m=+17.043856340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:24.973491 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:24.973441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:24.973668 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.973601    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:24.973723 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:24.973677    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:32.973658641 +0000 UTC m=+17.144733237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:25.319725 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:25.319644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:25.319903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:25.319644    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:25.319903 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:25.319808    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:25.319903 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:25.319860    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:26.320235 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:26.320196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:26.320760 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:26.320340    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:27.319621 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:27.319581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:27.319621 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:27.319620    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:27.319852 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:27.319713    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:27.319921 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:27.319848    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:28.319635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:28.319596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:28.320103 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:28.319763    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:29.319617 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:29.319578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:29.319854 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:29.319578    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:29.319854 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:29.319708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:29.320257 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:29.319841    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:30.319367 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:30.319338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:30.319564 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:30.319461    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:31.320006 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:31.319962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:31.320006 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:31.319996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:31.320460 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:31.320105    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:31.320460 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:31.320246    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:32.319857 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:32.319830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:32.319993 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.319945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:32.931372 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:32.931336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:32.931392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931510    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931526    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931543    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931553    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931591    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:45:48.931569996 +0000 UTC m=+33.102644602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:32.931851 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:32.931612    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:45:48.931601744 +0000 UTC m=+33.102676348 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:33.032258 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:33.032219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:33.032438 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:33.032373    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:33.032438 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:33.032435    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:49.032418067 +0000 UTC m=+33.203492686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:33.319290 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:33.319209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:33.319290 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:33.319253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:33.319500 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:33.319344    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:33.319500 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:33.319461    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:34.319974 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:34.319940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:34.320528 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:34.320076    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:35.319912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:35.319871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:35.320100 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:35.319871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:35.320100 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:35.320002    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:35.320430 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:35.320105    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:36.320549 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.320522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:36.321124 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:36.320639    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:36.549723 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.549439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal" event={"ID":"e90d483920ce499f7a5df446cbd8da0f","Type":"ContainerStarted","Data":"d4f72c30a9df6384d9f918a435c91b2979c9dfc7eacc2e37b38b02cbd2967fe5"}
Sep 04 22:45:36.551168 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.551110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxqcj" event={"ID":"4fb30bf8-0f7f-4a15-9d09-3db3ee1b436d","Type":"ContainerStarted","Data":"29bb52ec793c62501c65a5b26e0c0753c5e520630f2e40cb8572a32564893bc8"}
Sep 04 22:45:36.552715 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.552694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qhmhp" event={"ID":"7ca99630-4b48-417c-aff0-efe8419fdb18","Type":"ContainerStarted","Data":"bb460d0993befd2d50b60e4047ec98984569ec9dfbcebaaca7df9717aaaaacb2"}
Sep 04 22:45:36.562882 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.562834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-172.ec2.internal" podStartSLOduration=19.562821625 podStartE2EDuration="19.562821625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:36.561964404 +0000 UTC m=+20.733039017" watchObservedRunningTime="2026-09-04 22:45:36.562821625 +0000 UTC m=+20.733896237"
Sep 04 22:45:36.595142 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.595094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qhmhp" podStartSLOduration=1.869687897 podStartE2EDuration="20.595075712s" totalImagesPullingTime="18.725387815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:36.577369165 +0000 UTC m=+20.748443772" watchObservedRunningTime="2026-09-04 22:45:36.595075712 +0000 UTC m=+20.766150570"
Sep 04 22:45:36.595382 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:36.595364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bxqcj" podStartSLOduration=1.943721316 podStartE2EDuration="20.595360099s" totalImagesPullingTime="18.651638783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:36.59504632 +0000 UTC m=+20.766120943" watchObservedRunningTime="2026-09-04 22:45:36.595360099 +0000 UTC m=+20.766434747"
Sep 04 22:45:37.320259 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.320062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:37.320401 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.320144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:37.320401 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:37.320328    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:37.320495 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:37.320411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:37.560458 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.560418    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="24d85264db306aa67ad2e20f064e53621f31db4d6bbb1efe7e5428827d0ed599" exitCode=0
Sep 04 22:45:37.561228 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.560462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"24d85264db306aa67ad2e20f064e53621f31db4d6bbb1efe7e5428827d0ed599"}
Sep 04 22:45:37.564225 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.562421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv" event={"ID":"3169f0f2-0259-4375-a406-b14f54840ff2","Type":"ContainerStarted","Data":"809b9cd5ee28fc4d14005b418f927b4fc3b13f9a131b65373efe8577918d76d7"}
Sep 04 22:45:37.566356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.566328    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7465875ab6954932567f18fa44a4de7" containerID="a53006a95b9de413dd4cafbc10009deb852baeb551895c3306fd5f0d5854b8e2" exitCode=0
Sep 04 22:45:37.566485 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.566406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal" event={"ID":"e7465875ab6954932567f18fa44a4de7","Type":"ContainerDied","Data":"a53006a95b9de413dd4cafbc10009deb852baeb551895c3306fd5f0d5854b8e2"}
Sep 04 22:45:37.569785 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.569763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"e5fc403ea4a28ebcab5a0e7e002fd2a73890ae64142e7d27c501965cd981cb09"}
Sep 04 22:45:37.569895 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.569864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"6814b1f83fbb5215c20ef3efa5b2da0f63955ba8232cce9cc40059ea587271e1"}
Sep 04 22:45:37.569943 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.569908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"48a6c8542950e8c313352165d1d15699dfcf5db63b5e3a70239a96c612b58c2a"}
Sep 04 22:45:37.569943 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.569938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"c4e11967fdc179c94fcc74b461215049cf4d4197a35e2d049522aab4d7674bcd"}
Sep 04 22:45:37.570023 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.569965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"2eca4dfda569cdfd08db3875c6d8be136bc9c2727fb8800624f7e3ecca61ecc7"}
Sep 04 22:45:37.570023 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.570007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"34cb0ad9f596522d86a4cb68ccb029e40419bf7c6ce0dc01ca5002972fa7689e"}
Sep 04 22:45:37.571176 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.571153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m424m" event={"ID":"4ee6f1db-453b-4985-aa29-774fd08ba876","Type":"ContainerStarted","Data":"9e00cfc47e17e0c0650b1da4ec8d0be14f24edb73424be54fa4d05ac37a62a18"}
Sep 04 22:45:37.572497 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.572478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nhj5d" event={"ID":"de5fdc91-819b-4568-bb01-54547f4735cd","Type":"ContainerStarted","Data":"d596b9643214a85ce0eae9cfb6352e3eebc154802873a4277d703f4b78745af5"}
Sep 04 22:45:37.573757 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.573721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxmgs" event={"ID":"7cef06c4-9dc5-4510-96de-855b61f090fb","Type":"ContainerStarted","Data":"61241e1d912bb05127484419004543d8b456e978206df27d7406586123ee1386"}
Sep 04 22:45:37.612637 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.612383    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxmgs" podStartSLOduration=2.875609274 podStartE2EDuration="21.612366452s" totalImagesPullingTime="18.736757178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:37.612080768 +0000 UTC m=+21.783155380" watchObservedRunningTime="2026-09-04 22:45:37.612366452 +0000 UTC m=+21.783441086"
Sep 04 22:45:37.626489 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.626437    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nhj5d" podStartSLOduration=2.942429591 podStartE2EDuration="21.626416201s" totalImagesPullingTime="18.68398661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:37.625676791 +0000 UTC m=+21.796751404" watchObservedRunningTime="2026-09-04 22:45:37.626416201 +0000 UTC m=+21.797490814"
Sep 04 22:45:37.639908 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:37.639857    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m424m" podStartSLOduration=3.032119824 podStartE2EDuration="21.639839806s" totalImagesPullingTime="18.607719982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:37.639338312 +0000 UTC m=+21.810412923" watchObservedRunningTime="2026-09-04 22:45:37.639839806 +0000 UTC m=+21.810914418"
Sep 04 22:45:38.199445 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.199394    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 04 22:45:38.268749 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.268566    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-04T22:45:38.199420765Z","UUID":"1bb1ff46-a0f4-4536-a08e-579c4711e1b0","Handler":null,"Name":"","Endpoint":""}
Sep 04 22:45:38.270288 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.270265    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 04 22:45:38.270424 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.270298    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 04 22:45:38.320354 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.320323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:38.320533 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:38.320475    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:38.578126 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.578037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kgdwx" event={"ID":"19fe17dd-bddb-4c93-ae78-72d7a45694b8","Type":"ContainerStarted","Data":"2769c25e7798d96a35aa8029142c87198656c1fa93b686310477e1343a42c451"}
Sep 04 22:45:38.580270 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.580241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv" event={"ID":"3169f0f2-0259-4375-a406-b14f54840ff2","Type":"ContainerStarted","Data":"373d4f17aaaf10f19e883ba2772f720b995cdb4b4a1fad90adbf8d42eef562c8"}
Sep 04 22:45:38.582208 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.582182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal" event={"ID":"e7465875ab6954932567f18fa44a4de7","Type":"ContainerStarted","Data":"602956659ea96d8838bd2f20ad509eb79ab61181d4180f2e4d909b3d12a2332d"}
Sep 04 22:45:38.591579 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.591518    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kgdwx" podStartSLOduration=3.942279124 podStartE2EDuration="22.591497294s" totalImagesPullingTime="18.64921817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:38.591389035 +0000 UTC m=+22.762463649" watchObservedRunningTime="2026-09-04 22:45:38.591497294 +0000 UTC m=+22.762571911"
Sep 04 22:45:38.607782 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:38.606659    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-172.ec2.internal" podStartSLOduration=21.606639902 podStartE2EDuration="21.606639902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:38.605444578 +0000 UTC m=+22.776519189" watchObservedRunningTime="2026-09-04 22:45:38.606639902 +0000 UTC m=+22.777714515"
Sep 04 22:45:39.319974 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:39.319940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:39.320232 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:39.319942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:39.320232 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:39.320048    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:39.320232 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:39.320119    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:39.587717 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:39.587483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv" event={"ID":"3169f0f2-0259-4375-a406-b14f54840ff2","Type":"ContainerStarted","Data":"9bdf68ceba426a35ebaedcc9a6dde6988c951df9d3f65cdd4498cd65bfc1eae5"}
Sep 04 22:45:39.607920 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:39.607857    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwpqv" podStartSLOduration=2.975369115 podStartE2EDuration="23.607839824s" totalImagesPullingTime="20.632470709s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:39.606257436 +0000 UTC m=+23.777332062" watchObservedRunningTime="2026-09-04 22:45:39.607839824 +0000 UTC m=+23.778914436"
Sep 04 22:45:40.320144 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:40.320100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:40.320359 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:40.320249    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:40.593506 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:40.593416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"55780b44849f39451569a513970cb539eda30f8b1797a5d301be073e5c2ca8d6"}
Sep 04 22:45:41.320274 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:41.320236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:41.320471 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:41.320236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:41.320471 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:41.320346    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:41.320471 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:41.320447    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:41.840124 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:41.840093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:41.840718 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:41.840700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:42.319941 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.319702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:42.320146 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:42.319990    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:42.599990 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.599901    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="aacc232a186884b1df5680f76a94f65fb6db5705d7cd7b1c175531c58a821edf" exitCode=0
Sep 04 22:45:42.600152 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.599982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"aacc232a186884b1df5680f76a94f65fb6db5705d7cd7b1c175531c58a821edf"}
Sep 04 22:45:42.604066 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.604027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" event={"ID":"21916f8f-fde8-475e-bbfc-c443fcccbe80","Type":"ContainerStarted","Data":"d87acc5dc5a597ece8262cebfa2fa552ac0a0fe98a4f4da8423cb29cd0096a69"}
Sep 04 22:45:42.604397 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.604373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:42.604828 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.604811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxmgs"
Sep 04 22:45:42.646117 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:42.646061    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd" podStartSLOduration=7.470586554 podStartE2EDuration="26.64604503s" totalImagesPullingTime="19.175458476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:42.645318988 +0000 UTC m=+26.816393601" watchObservedRunningTime="2026-09-04 22:45:42.64604503 +0000 UTC m=+26.817119700"
Sep 04 22:45:43.319991 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.319956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:43.319991 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.319988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:43.320547 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:43.320113    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:43.320547 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:43.320277    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:43.606238 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.606150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:43.606238 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.606235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:43.606647 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.606622    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:43.621854 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.621828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:43.622067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.621967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:45:43.701450 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.701422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mcpvb"]
Sep 04 22:45:43.701609 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.701593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:43.701729 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:43.701708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:43.704339 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.704309    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c75hg"]
Sep 04 22:45:43.704438 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.704419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:43.704538 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:43.704519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:43.705138 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.705113    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nh76j"]
Sep 04 22:45:43.705239 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:43.705214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:43.705325 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:43.705302    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:44.610061 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:44.610027    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="2e8765d56a0801b7f80d2e47fcab04d9a92c54ff611f59271149f0e1725a3f2b" exitCode=0
Sep 04 22:45:44.610820 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:44.610110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"2e8765d56a0801b7f80d2e47fcab04d9a92c54ff611f59271149f0e1725a3f2b"}
Sep 04 22:45:45.319331 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:45.319303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:45.319493 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:45.319405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:45.319493 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:45.319432    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:45.319568 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:45.319499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:45.319568 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:45.319534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:45.319631 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:45.319579    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:46.617159 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:46.616886    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="d616829c6970f56c712bc8517bc77360a6aa17cb2350448dfb75a9745981087e" exitCode=0
Sep 04 22:45:46.617159 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:46.616974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"d616829c6970f56c712bc8517bc77360a6aa17cb2350448dfb75a9745981087e"}
Sep 04 22:45:47.320308 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:47.320273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:47.320515 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:47.320321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:47.320515 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:47.320321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:47.320515 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:47.320411    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:47.320669 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:47.320519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:47.320669 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:47.320626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:48.947867 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:48.947826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:48.947884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.947998    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.948050    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:46:20.948036681 +0000 UTC m=+65.119111271 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.948047    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.948074    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.948089    2577 projected.go:196] Error preparing data for projected volume kube-api-access-ldb28 for pod openshift-network-diagnostics/network-check-target-nh76j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:48.948351 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:48.948163    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28 podName:d5424a3b-48b2-42e4-be05-e476ab369cee nodeName:}" failed. No retries permitted until 2026-09-04 22:46:20.948144425 +0000 UTC m=+65.119219022 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ldb28" (UniqueName: "kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28") pod "network-check-target-nh76j" (UID: "d5424a3b-48b2-42e4-be05-e476ab369cee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:45:49.049151 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:49.049114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:49.049296 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:49.049228    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:49.049296 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:49.049293    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret podName:c1fcc2d0-572d-4bc6-91bc-44a34539f922 nodeName:}" failed. No retries permitted until 2026-09-04 22:46:21.049278994 +0000 UTC m=+65.220353585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret") pod "global-pull-secret-syncer-c75hg" (UID: "c1fcc2d0-572d-4bc6-91bc-44a34539f922") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:45:49.319786 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:49.319686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:49.319947 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:49.319686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:49.319947 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:49.319820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:45:49.319947 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:49.319686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:49.319947 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:49.319892    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c75hg" podUID="c1fcc2d0-572d-4bc6-91bc-44a34539f922"
Sep 04 22:45:49.320154 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:49.319966    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nh76j" podUID="d5424a3b-48b2-42e4-be05-e476ab369cee"
Sep 04 22:45:50.601955 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.601883    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-172.ec2.internal" event="NodeReady"
Sep 04 22:45:50.602295 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.602028    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 04 22:45:50.645964 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.645933    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6pmhs"]
Sep 04 22:45:50.660084 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.660047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mjcjb"]
Sep 04 22:45:50.660270 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.660181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.662653 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.662621    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 04 22:45:50.662801 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.662689    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 04 22:45:50.662801 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.662697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cv6t6\""
Sep 04 22:45:50.672227 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.672201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pmhs"]
Sep 04 22:45:50.672377 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.672245    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjcjb"]
Sep 04 22:45:50.672377 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.672312    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.675151 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.675130    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 04 22:45:50.675261 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.675193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w24qj\""
Sep 04 22:45:50.675261 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.675196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 04 22:45:50.675460 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.675445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 04 22:45:50.762004 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.761957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0618c88-28ff-4047-b6a4-36a3088c7e49-cert\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.762177 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.762017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qpr\" (UniqueName: \"kubernetes.io/projected/217286bb-2071-42aa-b834-bdf9a1b3ad85-kube-api-access-44qpr\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.762177 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.762061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgws\" (UniqueName: \"kubernetes.io/projected/b0618c88-28ff-4047-b6a4-36a3088c7e49-kube-api-access-ktgws\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.762177 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.762078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/217286bb-2071-42aa-b834-bdf9a1b3ad85-config-volume\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.762177 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.762098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.762177 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.762123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217286bb-2071-42aa-b834-bdf9a1b3ad85-tmp-dir\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863238 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863238 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217286bb-2071-42aa-b834-bdf9a1b3ad85-tmp-dir\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863238 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0618c88-28ff-4047-b6a4-36a3088c7e49-cert\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.863505 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:50.863315    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:45:50.863505 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44qpr\" (UniqueName: \"kubernetes.io/projected/217286bb-2071-42aa-b834-bdf9a1b3ad85-kube-api-access-44qpr\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863505 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:50.863396    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:51.363365172 +0000 UTC m=+35.534439785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:45:50.863505 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgws\" (UniqueName: \"kubernetes.io/projected/b0618c88-28ff-4047-b6a4-36a3088c7e49-kube-api-access-ktgws\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.863505 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/217286bb-2071-42aa-b834-bdf9a1b3ad85-config-volume\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863771 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/217286bb-2071-42aa-b834-bdf9a1b3ad85-tmp-dir\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.863958 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.863942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/217286bb-2071-42aa-b834-bdf9a1b3ad85-config-volume\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.867366 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.867346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0618c88-28ff-4047-b6a4-36a3088c7e49-cert\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.872765 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.872723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgws\" (UniqueName: \"kubernetes.io/projected/b0618c88-28ff-4047-b6a4-36a3088c7e49-kube-api-access-ktgws\") pod \"ingress-canary-mjcjb\" (UID: \"b0618c88-28ff-4047-b6a4-36a3088c7e49\") " pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:50.872988 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.872968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qpr\" (UniqueName: \"kubernetes.io/projected/217286bb-2071-42aa-b834-bdf9a1b3ad85-kube-api-access-44qpr\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:50.980789 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:50.980731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mjcjb"
Sep 04 22:45:51.304007 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.303760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mjcjb"]
Sep 04 22:45:51.315650 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:45:51.315615    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0618c88_28ff_4047_b6a4_36a3088c7e49.slice/crio-7a367795570b8455243322014da21e130c16c4fb8ec79fc1ca9ac99b165dcdc2 WatchSource:0}: Error finding container 7a367795570b8455243322014da21e130c16c4fb8ec79fc1ca9ac99b165dcdc2: Status 404 returned error can't find the container with id 7a367795570b8455243322014da21e130c16c4fb8ec79fc1ca9ac99b165dcdc2
Sep 04 22:45:51.319306 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.319284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:45:51.319415 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.319393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:45:51.319508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.319475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:45:51.322008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.321842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 22:45:51.322008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.321852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnghc\""
Sep 04 22:45:51.322008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.321865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 22:45:51.322008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.321906    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 22:45:51.322008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.321938    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 22:45:51.322255 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.322074    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxf76\""
Sep 04 22:45:51.368494 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.368463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:51.368646 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:51.368628    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:45:51.368704 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:51.368693    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:52.36867789 +0000 UTC m=+36.539752479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:45:51.631376 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.631342    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="0564c309f59c2645215fe066eb5443a7a571d2ad7a9a547d7923080281b33d27" exitCode=0
Sep 04 22:45:51.631929 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.631416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"0564c309f59c2645215fe066eb5443a7a571d2ad7a9a547d7923080281b33d27"}
Sep 04 22:45:51.632452 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:51.632434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjcjb" event={"ID":"b0618c88-28ff-4047-b6a4-36a3088c7e49","Type":"ContainerStarted","Data":"7a367795570b8455243322014da21e130c16c4fb8ec79fc1ca9ac99b165dcdc2"}
Sep 04 22:45:52.376972 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:52.376939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:52.377140 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:52.377094    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:45:52.377224 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:52.377157    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:54.377142137 +0000 UTC m=+38.548216727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:45:52.638991 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:52.638898    2577 generic.go:356] "Generic (PLEG): container finished" podID="959228e2-e29a-422b-af73-02a7fa386332" containerID="f530c6c575cb84fd8e2c1482fc4057f17c5261af925088c58d94ebae96b8d4ba" exitCode=0
Sep 04 22:45:52.638991 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:52.638963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerDied","Data":"f530c6c575cb84fd8e2c1482fc4057f17c5261af925088c58d94ebae96b8d4ba"}
Sep 04 22:45:53.645350 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:53.645266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" event={"ID":"959228e2-e29a-422b-af73-02a7fa386332","Type":"ContainerStarted","Data":"4d7be8b1e706ecf857d7073f24c5e3333c2c83e21b22b2a0d5cf841eb0e31009"}
Sep 04 22:45:53.666308 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:53.666251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m4fgs" podStartSLOduration=8.404096881 podStartE2EDuration="37.666230386s" totalImagesPullingTime="29.262133505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:53.666188309 +0000 UTC m=+37.837262945" watchObservedRunningTime="2026-09-04 22:45:53.666230386 +0000 UTC m=+37.837304999"
Sep 04 22:45:54.391733 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:54.391695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:54.391930 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:54.391844    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:45:54.391930 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:54.391906    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:45:58.391889999 +0000 UTC m=+42.562964588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:45:55.651254 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:55.651216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mjcjb" event={"ID":"b0618c88-28ff-4047-b6a4-36a3088c7e49","Type":"ContainerStarted","Data":"868c41e0a7f4ec35ee3e01f392c6289190149358bcc6976eb71aa564fa9641e0"}
Sep 04 22:45:55.667596 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:55.667545    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mjcjb" podStartSLOduration=2.244639246 podStartE2EDuration="5.667530675s" totalImagesPullingTime="3.422891429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:55.666283012 +0000 UTC m=+39.837357624" watchObservedRunningTime="2026-09-04 22:45:55.667530675 +0000 UTC m=+39.838605286"
Sep 04 22:45:58.422175 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:45:58.422136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:45:58.422561 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:58.422289    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:45:58.422561 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:45:58.422353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:46:06.422336685 +0000 UTC m=+50.593411276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:46:06.474581 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:06.474536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:46:06.475079 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:06.474649    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:46:06.475079 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:06.474709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:46:22.474695676 +0000 UTC m=+66.645770266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:46:15.622995 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:15.622957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p66rd"
Sep 04 22:46:20.980384 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:20.980337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:46:20.980815 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:20.980400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:46:20.983431 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:20.983412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 22:46:20.983476 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:20.983417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 22:46:20.991290 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:20.991263    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 04 22:46:20.991345 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:20.991335    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:47:24.991317314 +0000 UTC m=+129.162391908 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : secret "metrics-daemon-secret" not found
Sep 04 22:46:20.993211 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:20.993191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 22:46:21.005265 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.005234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldb28\" (UniqueName: \"kubernetes.io/projected/d5424a3b-48b2-42e4-be05-e476ab369cee-kube-api-access-ldb28\") pod \"network-check-target-nh76j\" (UID: \"d5424a3b-48b2-42e4-be05-e476ab369cee\") " pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:46:21.031301 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.031269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnghc\""
Sep 04 22:46:21.039215 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.039185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:46:21.080670 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.080629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:46:21.083640 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.083618    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 22:46:21.094068 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.094034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1fcc2d0-572d-4bc6-91bc-44a34539f922-original-pull-secret\") pod \"global-pull-secret-syncer-c75hg\" (UID: \"c1fcc2d0-572d-4bc6-91bc-44a34539f922\") " pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:46:21.325817 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.325724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nh76j"]
Sep 04 22:46:21.329078 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:46:21.329054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5424a3b_48b2_42e4_be05_e476ab369cee.slice/crio-afbf1c18a443a5f6b6d1ef2b76a998c7737966e5b186dfe00578ef52c681d910 WatchSource:0}: Error finding container afbf1c18a443a5f6b6d1ef2b76a998c7737966e5b186dfe00578ef52c681d910: Status 404 returned error can't find the container with id afbf1c18a443a5f6b6d1ef2b76a998c7737966e5b186dfe00578ef52c681d910
Sep 04 22:46:21.340469 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.340447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c75hg"
Sep 04 22:46:21.630555 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.630522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c75hg"]
Sep 04 22:46:21.634295 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:46:21.634269    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fcc2d0_572d_4bc6_91bc_44a34539f922.slice/crio-43502529a894867d1c5a10d5436bf7987766625d796662229fc726936593a244 WatchSource:0}: Error finding container 43502529a894867d1c5a10d5436bf7987766625d796662229fc726936593a244: Status 404 returned error can't find the container with id 43502529a894867d1c5a10d5436bf7987766625d796662229fc726936593a244
Sep 04 22:46:21.715900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.715860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nh76j" event={"ID":"d5424a3b-48b2-42e4-be05-e476ab369cee","Type":"ContainerStarted","Data":"afbf1c18a443a5f6b6d1ef2b76a998c7737966e5b186dfe00578ef52c681d910"}
Sep 04 22:46:21.716918 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:21.716899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c75hg" event={"ID":"c1fcc2d0-572d-4bc6-91bc-44a34539f922","Type":"ContainerStarted","Data":"43502529a894867d1c5a10d5436bf7987766625d796662229fc726936593a244"}
Sep 04 22:46:22.489650 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:22.489423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:46:22.489650 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:22.489569    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:46:22.489650 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:22.489643    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:46:54.489623255 +0000 UTC m=+98.660697846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:46:24.727117 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:24.727065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nh76j" event={"ID":"d5424a3b-48b2-42e4-be05-e476ab369cee","Type":"ContainerStarted","Data":"feff2c0457519e95d49a4a76cd71da46a8fcf2a9f0cf144fd185e06ef415f644"}
Sep 04 22:46:24.727586 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:24.727253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:46:24.741860 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:24.741798    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nh76j" podStartSLOduration=65.883839281 podStartE2EDuration="1m8.741778725s" totalImagesPullingTime="2.857939444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:46:24.741548775 +0000 UTC m=+68.912623388" watchObservedRunningTime="2026-09-04 22:46:24.741778725 +0000 UTC m=+68.912853339"
Sep 04 22:46:26.733310 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:26.733274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c75hg" event={"ID":"c1fcc2d0-572d-4bc6-91bc-44a34539f922","Type":"ContainerStarted","Data":"ea058a5a394a932aecc0d0968f2938fb02718f89a62aa3afbb7b9190fa553b47"}
Sep 04 22:46:26.772697 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:26.772639    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c75hg" podStartSLOduration=66.649519598 podStartE2EDuration="1m10.772621678s" totalImagesPullingTime="4.12310208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:46:26.771936013 +0000 UTC m=+70.943010625" watchObservedRunningTime="2026-09-04 22:46:26.772621678 +0000 UTC m=+70.943696284"
Sep 04 22:46:54.515386 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:54.515327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:46:54.515852 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:54.515459    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:46:54.515852 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:46:54.515552    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:47:58.515536776 +0000 UTC m=+162.686611365 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:46:55.732078 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:46:55.732047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nh76j"
Sep 04 22:47:02.069246 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.069206    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"]
Sep 04 22:47:02.072076 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.072058    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"]
Sep 04 22:47:02.072195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.072177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.074163 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.074139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.074754 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.074717    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 04 22:47:02.074867 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.074778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 04 22:47:02.074867 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.074786    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 04 22:47:02.074867 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.074792    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 04 22:47:02.076828 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.076803    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 04 22:47:02.077258 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.077234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 04 22:47:02.077535 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.077510    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 04 22:47:02.077803 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.077784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 04 22:47:02.082697 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.082673    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"]
Sep 04 22:47:02.083408 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.083389    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"]
Sep 04 22:47:02.162845 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.162800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bbb3305-e788-4801-8887-c0e71539b666-tmp\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.162851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.162883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qg5\" (UniqueName: \"kubernetes.io/projected/4bbb3305-e788-4801-8887-c0e71539b666-kube-api-access-n8qg5\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.162931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4924cb65-f77e-437e-832e-fb749d30e111-additional-service-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.162972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.163011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4bbb3305-e788-4801-8887-c0e71539b666-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.163044 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.163031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.163313 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.163096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frh62\" (UniqueName: \"kubernetes.io/projected/4924cb65-f77e-437e-832e-fb749d30e111-kube-api-access-frh62\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.163313 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.163132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.263937 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.263895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.263965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qg5\" (UniqueName: \"kubernetes.io/projected/4bbb3305-e788-4801-8887-c0e71539b666-kube-api-access-n8qg5\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.264195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.263998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4924cb65-f77e-437e-832e-fb749d30e111-additional-service-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4bbb3305-e788-4801-8887-c0e71539b666-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.264195 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264514 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frh62\" (UniqueName: \"kubernetes.io/projected/4924cb65-f77e-437e-832e-fb749d30e111-kube-api-access-frh62\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264514 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.264514 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bbb3305-e788-4801-8887-c0e71539b666-tmp\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.264712 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bbb3305-e788-4801-8887-c0e71539b666-tmp\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.264798 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.264729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4924cb65-f77e-437e-832e-fb749d30e111-additional-service-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.266548 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.266522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.266806 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.266772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.266920 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.266780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4bbb3305-e788-4801-8887-c0e71539b666-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.267054 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.267032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-hub\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.267502 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.267484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4924cb65-f77e-437e-832e-fb749d30e111-ca\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.272623 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.272594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frh62\" (UniqueName: \"kubernetes.io/projected/4924cb65-f77e-437e-832e-fb749d30e111-kube-api-access-frh62\") pod \"cluster-proxy-proxy-agent-845bc7bcb4-gfvtg\" (UID: \"4924cb65-f77e-437e-832e-fb749d30e111\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.272719 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.272656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qg5\" (UniqueName: \"kubernetes.io/projected/4bbb3305-e788-4801-8887-c0e71539b666-kube-api-access-n8qg5\") pod \"klusterlet-addon-workmgr-b8b96959b-tbhpf\" (UID: \"4bbb3305-e788-4801-8887-c0e71539b666\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.386273 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.386236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:02.400146 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.400097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:47:02.657152 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.657060    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"]
Sep 04 22:47:02.660230 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:47:02.660195    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbb3305_e788_4801_8887_c0e71539b666.slice/crio-ab6148a6efc535fe8f977c480d1485996fc2c65884d245dc8f5b1f0e2b6fe033 WatchSource:0}: Error finding container ab6148a6efc535fe8f977c480d1485996fc2c65884d245dc8f5b1f0e2b6fe033: Status 404 returned error can't find the container with id ab6148a6efc535fe8f977c480d1485996fc2c65884d245dc8f5b1f0e2b6fe033
Sep 04 22:47:02.679815 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.679789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"]
Sep 04 22:47:02.682928 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:47:02.682905    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4924cb65_f77e_437e_832e_fb749d30e111.slice/crio-95a61e04d3fd1d7155333eec206b2c1d7ecb77821d26018a03e347ec86a03f45 WatchSource:0}: Error finding container 95a61e04d3fd1d7155333eec206b2c1d7ecb77821d26018a03e347ec86a03f45: Status 404 returned error can't find the container with id 95a61e04d3fd1d7155333eec206b2c1d7ecb77821d26018a03e347ec86a03f45
Sep 04 22:47:02.818043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.818003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerStarted","Data":"95a61e04d3fd1d7155333eec206b2c1d7ecb77821d26018a03e347ec86a03f45"}
Sep 04 22:47:02.819039 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:02.819014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" event={"ID":"4bbb3305-e788-4801-8887-c0e71539b666","Type":"ContainerStarted","Data":"ab6148a6efc535fe8f977c480d1485996fc2c65884d245dc8f5b1f0e2b6fe033"}
Sep 04 22:47:08.837294 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.837259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" event={"ID":"4bbb3305-e788-4801-8887-c0e71539b666","Type":"ContainerStarted","Data":"7901c49f11ee5b23eb934b5fe6545c38ddf3eae13583114842ecdfa87c556d7c"}
Sep 04 22:47:08.837754 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.837390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:08.839229 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.839211    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:47:08.839479 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.839460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerStarted","Data":"c546c87cdce43c23697f1f738e84056022ac52ea35a9fc25ea64d4fc68e4927f"}
Sep 04 22:47:08.839543 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.839492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerStarted","Data":"e78f466bf9cbc5e88eb4e18118766d2efaa70b4bad73db10cdcf4c61008f9591"}
Sep 04 22:47:08.839543 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.839504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerStarted","Data":"efcaa1f747d22acce76d794d99c19bc89b62bb6c2ad052611673a5b2212e129b"}
Sep 04 22:47:08.854174 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.854132    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" podStartSLOduration=1.185708364 podStartE2EDuration="6.85411714s" totalImagesPullingTime="5.668408776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:47:08.852493491 +0000 UTC m=+113.023568103" watchObservedRunningTime="2026-09-04 22:47:08.85411714 +0000 UTC m=+113.025191751"
Sep 04 22:47:08.869415 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:08.869375    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" podStartSLOduration=1.220793409 podStartE2EDuration="6.86936179s" totalImagesPullingTime="5.648568381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:47:08.8688295 +0000 UTC m=+113.039904112" watchObservedRunningTime="2026-09-04 22:47:08.86936179 +0000 UTC m=+113.040436402"
Sep 04 22:47:25.034617 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:25.034566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:47:25.035241 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:25.034762    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 04 22:47:25.035241 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:25.034851    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs podName:0333a425-3e65-479b-a1b6-2419b0d01dbe nodeName:}" failed. No retries permitted until 2026-09-04 22:49:27.0348272 +0000 UTC m=+251.205901803 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs") pod "network-metrics-daemon-mcpvb" (UID: "0333a425-3e65-479b-a1b6-2419b0d01dbe") : secret "metrics-daemon-secret" not found
Sep 04 22:47:53.670533 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:53.670483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6pmhs" podUID="217286bb-2071-42aa-b834-bdf9a1b3ad85"
Sep 04 22:47:53.964042 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:53.963968    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:47:54.335623 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:54.335595    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mcpvb" podUID="0333a425-3e65-479b-a1b6-2419b0d01dbe"
Sep 04 22:47:58.561932 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:47:58.561891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:47:58.562416 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:58.562058    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:47:58.562416 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:47:58.562135    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls podName:217286bb-2071-42aa-b834-bdf9a1b3ad85 nodeName:}" failed. No retries permitted until 2026-09-04 22:50:00.562117135 +0000 UTC m=+284.733191726 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls") pod "dns-default-6pmhs" (UID: "217286bb-2071-42aa-b834-bdf9a1b3ad85") : secret "dns-default-metrics-tls" not found
Sep 04 22:48:09.319578 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:09.319482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:48:58.811034 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.810997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-shfj2"]
Sep 04 22:48:58.812860 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.812844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.815459 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.815437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 04 22:48:58.815641 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.815627    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 04 22:48:58.816333 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.816313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 04 22:48:58.816426 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.816351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 04 22:48:58.816426 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.816357    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbtwn\""
Sep 04 22:48:58.825875 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.825851    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shfj2"]
Sep 04 22:48:58.878557 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.878528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a175609-ce32-450b-9ac1-7760ff1956a8-crio-socket\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.878557 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.878559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.878790 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.878603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwxh\" (UniqueName: \"kubernetes.io/projected/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-api-access-5qwxh\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.878790 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.878662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a175609-ce32-450b-9ac1-7760ff1956a8-data-volume\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.878790 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.878677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979532 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwxh\" (UniqueName: \"kubernetes.io/projected/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-api-access-5qwxh\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a175609-ce32-450b-9ac1-7760ff1956a8-data-volume\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a175609-ce32-450b-9ac1-7760ff1956a8-crio-socket\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:58.979689    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:48:58.979727 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a175609-ce32-450b-9ac1-7760ff1956a8-crio-socket\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.980016 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:58.979691    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:48:58.980016 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:58.979805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:48:59.479783608 +0000 UTC m=+223.650858206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : configmap "kube-rbac-proxy" not found
Sep 04 22:48:58.980016 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:58.979826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:48:59.47981665 +0000 UTC m=+223.650891241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:48:58.980016 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.979951    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a175609-ce32-450b-9ac1-7760ff1956a8-data-volume\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:58.987937 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:58.987915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwxh\" (UniqueName: \"kubernetes.io/projected/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-api-access-5qwxh\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:59.483825 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:59.483773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:59.483825 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:48:59.483828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:48:59.484070 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:59.483896    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:48:59.484070 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:59.483915    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:48:59.484070 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:59.483963    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:00.483947283 +0000 UTC m=+224.655021873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : configmap "kube-rbac-proxy" not found
Sep 04 22:48:59.484070 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:48:59.483977    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:00.483970786 +0000 UTC m=+224.655045376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:00.492076 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:00.492031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:00.492464 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:00.492143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:00.492464 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:00.492200    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:00.492464 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:00.492211    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:49:00.492464 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:00.492279    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:02.492259302 +0000 UTC m=+226.663333892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : configmap "kube-rbac-proxy" not found
Sep 04 22:49:00.492464 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:00.492294    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:02.492286364 +0000 UTC m=+226.663360954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:01.766594 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:01.766563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhj5d_de5fdc91-819b-4568-bb01-54547f4735cd/dns-node-resolver/0.log"
Sep 04 22:49:02.507315 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:02.507276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:02.507496 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:02.507355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:02.507496 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:02.507426    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:02.507573 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:02.507500    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:06.507482512 +0000 UTC m=+230.678557103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:02.507848 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:02.507832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a175609-ce32-450b-9ac1-7760ff1956a8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:02.566269 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:02.566238    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m424m_4ee6f1db-453b-4985-aa29-774fd08ba876/node-ca/0.log"
Sep 04 22:49:03.366712 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:03.366681    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjcjb_b0618c88-28ff-4047-b6a4-36a3088c7e49/serve-healthcheck-canary/0.log"
Sep 04 22:49:06.539089 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:06.539034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:06.539490 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:06.539191    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:06.539490 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:06.539262    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls podName:0a175609-ce32-450b-9ac1-7760ff1956a8 nodeName:}" failed. No retries permitted until 2026-09-04 22:49:14.539242073 +0000 UTC m=+238.710316663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-shfj2" (UID: "0a175609-ce32-450b-9ac1-7760ff1956a8") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:49:08.838326 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:08.838272    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" podUID="4bbb3305-e788-4801-8887-c0e71539b666" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.8:8000/readyz\": dial tcp 10.134.0.8:8000: connect: connection refused"
Sep 04 22:49:09.186519 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:09.186428    2577 generic.go:356] "Generic (PLEG): container finished" podID="4bbb3305-e788-4801-8887-c0e71539b666" containerID="7901c49f11ee5b23eb934b5fe6545c38ddf3eae13583114842ecdfa87c556d7c" exitCode=1
Sep 04 22:49:09.186519 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:09.186480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" event={"ID":"4bbb3305-e788-4801-8887-c0e71539b666","Type":"ContainerDied","Data":"7901c49f11ee5b23eb934b5fe6545c38ddf3eae13583114842ecdfa87c556d7c"}
Sep 04 22:49:09.186805 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:09.186792    2577 scope.go:117] "RemoveContainer" containerID="7901c49f11ee5b23eb934b5fe6545c38ddf3eae13583114842ecdfa87c556d7c"
Sep 04 22:49:10.191333 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:10.191303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf" event={"ID":"4bbb3305-e788-4801-8887-c0e71539b666","Type":"ContainerStarted","Data":"d20f8bb64f1ec09e17bfedc7ad9beec30176ba6347cdfd6d38fa47304e489e4f"}
Sep 04 22:49:10.191759 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:10.191547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:49:10.192203 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:10.192185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8b96959b-tbhpf"
Sep 04 22:49:14.597555 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:14.597522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:14.599785 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:14.599764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a175609-ce32-450b-9ac1-7760ff1956a8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-shfj2\" (UID: \"0a175609-ce32-450b-9ac1-7760ff1956a8\") " pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:14.721638 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:14.721589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-shfj2"
Sep 04 22:49:14.987529 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:14.987452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-shfj2"]
Sep 04 22:49:14.990789 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:14.990764    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a175609_ce32_450b_9ac1_7760ff1956a8.slice/crio-4fa063739b5e38e25d344c8d2f723afd36eb73a0f8012f21a77b722195c0e5b9 WatchSource:0}: Error finding container 4fa063739b5e38e25d344c8d2f723afd36eb73a0f8012f21a77b722195c0e5b9: Status 404 returned error can't find the container with id 4fa063739b5e38e25d344c8d2f723afd36eb73a0f8012f21a77b722195c0e5b9
Sep 04 22:49:15.208419 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:15.208378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shfj2" event={"ID":"0a175609-ce32-450b-9ac1-7760ff1956a8","Type":"ContainerStarted","Data":"d70b8b2f777f0e9538de943a4fef1f99f87ef6f9d38119913e4de1ba0a251457"}
Sep 04 22:49:15.208579 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:15.208436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shfj2" event={"ID":"0a175609-ce32-450b-9ac1-7760ff1956a8","Type":"ContainerStarted","Data":"4fa063739b5e38e25d344c8d2f723afd36eb73a0f8012f21a77b722195c0e5b9"}
Sep 04 22:49:16.212358 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:16.212321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shfj2" event={"ID":"0a175609-ce32-450b-9ac1-7760ff1956a8","Type":"ContainerStarted","Data":"f2b21a233b668186d589aa5b19a9d12a545f05578b0bba5ad3c8458e8fade4e1"}
Sep 04 22:49:17.217453 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:17.217407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-shfj2" event={"ID":"0a175609-ce32-450b-9ac1-7760ff1956a8","Type":"ContainerStarted","Data":"15d134b73a9c7454b80574709f406cb4f6e887a869e49ca090302fa4324326d4"}
Sep 04 22:49:17.235537 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:17.235483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-shfj2" podStartSLOduration=17.257165953 podStartE2EDuration="19.235467428s" totalImagesPullingTime="1.978301475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:48:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:17.234871996 +0000 UTC m=+241.405946618" watchObservedRunningTime="2026-09-04 22:49:17.235467428 +0000 UTC m=+241.406542040"
Sep 04 22:49:23.029355 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.029311    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85c9975d4d-dgwj2"]
Sep 04 22:49:23.031449 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.031425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.033805 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.033783    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 04 22:49:23.033945 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.033925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6zcvv\""
Sep 04 22:49:23.034685 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.034667    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 04 22:49:23.034758 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.034700    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 04 22:49:23.039651 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.039629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 04 22:49:23.048127 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.048095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c9975d4d-dgwj2"]
Sep 04 22:49:23.157466 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-bound-sa-token\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157466 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-installation-pull-secrets\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-certificates\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kpz\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-kube-api-access-65kpz\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-image-registry-private-configuration\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-tls\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-ca-trust-extracted\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.157687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.157631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-trusted-ca\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.257964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-bound-sa-token\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-installation-pull-secrets\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-certificates\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kpz\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-kube-api-access-65kpz\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-image-registry-private-configuration\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-tls\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-ca-trust-extracted\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258309 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-trusted-ca\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.258660 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-ca-trust-extracted\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.259019 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.258997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-certificates\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.259102 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.259037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-trusted-ca\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.260613 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.260579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-installation-pull-secrets\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.260924 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.260906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-registry-tls\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.260996 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.260975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-image-registry-private-configuration\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.265877 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.265852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-bound-sa-token\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.265997 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.265927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kpz\" (UniqueName: \"kubernetes.io/projected/50fcc1b1-f4ef-4b49-a24d-6b18a3318353-kube-api-access-65kpz\") pod \"image-registry-85c9975d4d-dgwj2\" (UID: \"50fcc1b1-f4ef-4b49-a24d-6b18a3318353\") " pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.340253 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.340219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:23.610528 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:23.610452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85c9975d4d-dgwj2"]
Sep 04 22:49:23.614206 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:23.614171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fcc1b1_f4ef_4b49_a24d_6b18a3318353.slice/crio-e3df6922f79ef73f6aab02aec433d76c7a96057d3a506b426e26c40035bf997c WatchSource:0}: Error finding container e3df6922f79ef73f6aab02aec433d76c7a96057d3a506b426e26c40035bf997c: Status 404 returned error can't find the container with id e3df6922f79ef73f6aab02aec433d76c7a96057d3a506b426e26c40035bf997c
Sep 04 22:49:24.239776 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:24.239725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" event={"ID":"50fcc1b1-f4ef-4b49-a24d-6b18a3318353","Type":"ContainerStarted","Data":"31f8077f1a7f841cf9524c6292efc21febdab7e6e3a3518073d084c9257c5bed"}
Sep 04 22:49:24.240158 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:24.239796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" event={"ID":"50fcc1b1-f4ef-4b49-a24d-6b18a3318353","Type":"ContainerStarted","Data":"e3df6922f79ef73f6aab02aec433d76c7a96057d3a506b426e26c40035bf997c"}
Sep 04 22:49:24.240158 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:24.239842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:49:24.259906 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:24.259833    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podStartSLOduration=1.259816722 podStartE2EDuration="1.259816722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:24.258136069 +0000 UTC m=+248.429210681" watchObservedRunningTime="2026-09-04 22:49:24.259816722 +0000 UTC m=+248.430891334"
Sep 04 22:49:27.084486 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:27.084440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:49:27.086795 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:27.086768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0333a425-3e65-479b-a1b6-2419b0d01dbe-metrics-certs\") pod \"network-metrics-daemon-mcpvb\" (UID: \"0333a425-3e65-479b-a1b6-2419b0d01dbe\") " pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:49:27.323344 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:27.323312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sxf76\""
Sep 04 22:49:27.331134 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:27.331107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mcpvb"
Sep 04 22:49:27.599014 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:27.598990    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mcpvb"]
Sep 04 22:49:27.601368 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:27.601338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0333a425_3e65_479b_a1b6_2419b0d01dbe.slice/crio-b8b7bdc3179d9098dbe73c9cd79d483fe51e0526d5f20fc44def4774c240a071 WatchSource:0}: Error finding container b8b7bdc3179d9098dbe73c9cd79d483fe51e0526d5f20fc44def4774c240a071: Status 404 returned error can't find the container with id b8b7bdc3179d9098dbe73c9cd79d483fe51e0526d5f20fc44def4774c240a071
Sep 04 22:49:28.254553 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:28.254523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mcpvb" event={"ID":"0333a425-3e65-479b-a1b6-2419b0d01dbe","Type":"ContainerStarted","Data":"b8b7bdc3179d9098dbe73c9cd79d483fe51e0526d5f20fc44def4774c240a071"}
Sep 04 22:49:29.259325 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:29.259282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mcpvb" event={"ID":"0333a425-3e65-479b-a1b6-2419b0d01dbe","Type":"ContainerStarted","Data":"7239742b8fde22e9bbe2b1f96a4c1cde23b81043c417801a986a4fbc909cf338"}
Sep 04 22:49:29.259713 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:29.259348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mcpvb" event={"ID":"0333a425-3e65-479b-a1b6-2419b0d01dbe","Type":"ContainerStarted","Data":"d83c772605627cfe7334145ad014fe4ab92d33a16f206d1c84e46bdd2682651b"}
Sep 04 22:49:29.274146 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:29.274097    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mcpvb" podStartSLOduration=252.494883324 podStartE2EDuration="4m13.274078925s" totalImagesPullingTime="779.195601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:29.273845311 +0000 UTC m=+253.444919922" watchObservedRunningTime="2026-09-04 22:49:29.274078925 +0000 UTC m=+253.445153538"
Sep 04 22:49:38.203220 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.203139    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jk8sd"]
Sep 04 22:49:38.206215 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.206193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.209436 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.209411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 04 22:49:38.209556 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.209448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 04 22:49:38.209556 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.209479    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 04 22:49:38.210428 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.210409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tvwbx\""
Sep 04 22:49:38.210501 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.210415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 04 22:49:38.210501 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.210482    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 04 22:49:38.210595 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.210510    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 04 22:49:38.364852 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.364808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-root\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365039 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.364863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365039 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.364941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxt5t\" (UniqueName: \"kubernetes.io/projected/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-kube-api-access-kxt5t\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365039 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-metrics-client-ca\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365039 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-tls\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365199 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-sys\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365199 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365199 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-textfile\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.365199 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.365120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-wtmp\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465620 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-root\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465620 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465620 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxt5t\" (UniqueName: \"kubernetes.io/projected/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-kube-api-access-kxt5t\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-metrics-client-ca\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-root\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-tls\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-sys\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-sys\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-textfile\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.465912 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.465895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-wtmp\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.466216 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.466060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-wtmp\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.466269 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.466218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-textfile\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.466333 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.466311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.466371 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.466311    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-metrics-client-ca\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.468197 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.468173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.468307 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.468196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-node-exporter-tls\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.473427 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.473404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxt5t\" (UniqueName: \"kubernetes.io/projected/18a578e3-edea-4e3d-ac1e-cd4dfb69bb39-kube-api-access-kxt5t\") pod \"node-exporter-jk8sd\" (UID: \"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39\") " pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.515567 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:38.515537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jk8sd"
Sep 04 22:49:38.526408 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:38.526375    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a578e3_edea_4e3d_ac1e_cd4dfb69bb39.slice/crio-aca32cac71fb43ae9edf8c3b842c883b074278b7ad9b9d93662c4b852f2e81d7 WatchSource:0}: Error finding container aca32cac71fb43ae9edf8c3b842c883b074278b7ad9b9d93662c4b852f2e81d7: Status 404 returned error can't find the container with id aca32cac71fb43ae9edf8c3b842c883b074278b7ad9b9d93662c4b852f2e81d7
Sep 04 22:49:39.292928 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:39.292893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk8sd" event={"ID":"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39","Type":"ContainerStarted","Data":"aca32cac71fb43ae9edf8c3b842c883b074278b7ad9b9d93662c4b852f2e81d7"}
Sep 04 22:49:40.297589 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:40.297554    2577 generic.go:356] "Generic (PLEG): container finished" podID="18a578e3-edea-4e3d-ac1e-cd4dfb69bb39" containerID="362e5e5a4fb47894e3e72efe32eac06b52319509eac3f8df69d52f04b5c7c976" exitCode=0
Sep 04 22:49:40.298102 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:40.297612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk8sd" event={"ID":"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39","Type":"ContainerDied","Data":"362e5e5a4fb47894e3e72efe32eac06b52319509eac3f8df69d52f04b5c7c976"}
Sep 04 22:49:41.302511 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:41.302475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk8sd" event={"ID":"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39","Type":"ContainerStarted","Data":"5dd8cbdc93621e4a09edffd8c9cf637476339e4f0df8f9acf198e5098e2b0dac"}
Sep 04 22:49:41.302887 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:41.302529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk8sd" event={"ID":"18a578e3-edea-4e3d-ac1e-cd4dfb69bb39","Type":"ContainerStarted","Data":"512e6f999e988b7873551a45d754c6154645fc6ff10f0b87ef8326eae6faf9c5"}
Sep 04 22:49:41.323255 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:41.323203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jk8sd" podStartSLOduration=2.355959979 podStartE2EDuration="3.323184928s" totalImagesPullingTime="905.462808ms" totalInitContainerRuntime="61.762141ms" isStatefulPod=true podCreationTimestamp="2026-09-04 22:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:41.322333931 +0000 UTC m=+265.493408543" watchObservedRunningTime="2026-09-04 22:49:41.323184928 +0000 UTC m=+265.494259540"
Sep 04 22:49:42.401349 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:42.401310    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" podUID="4924cb65-f77e-437e-832e-fb749d30e111" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 04 22:49:43.344884 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.344844    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:49:43.345068 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.344904    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:49:43.401789 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.401748    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-797975fdf9-7lnv6"]
Sep 04 22:49:43.405235 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.405213    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406560 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-serving-certs-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406673 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406673 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-metrics-client-ca\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406673 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406673 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406909 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf79\" (UniqueName: \"kubernetes.io/projected/79d2fdb6-f33f-435e-93aa-81785774bdd8-kube-api-access-8wf79\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406909 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-federate-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.406909 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.406836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.407640 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.407620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6wlwv\""
Sep 04 22:49:43.407758 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.407672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 04 22:49:43.407758 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.407686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 04 22:49:43.407758 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.407709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 04 22:49:43.408061 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.408044    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 04 22:49:43.408139 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.408044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 04 22:49:43.412646 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.412629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 04 22:49:43.418976 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.418957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-797975fdf9-7lnv6"]
Sep 04 22:49:43.507838 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.507797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.507853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf79\" (UniqueName: \"kubernetes.io/projected/79d2fdb6-f33f-435e-93aa-81785774bdd8-kube-api-access-8wf79\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.507892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-federate-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.507926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.507970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-serving-certs-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.508007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.508038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-metrics-client-ca\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508361 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.508061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508822 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.508751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-serving-certs-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.508963 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.508879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-metrics-client-ca\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.509033 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.509004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.510486 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.510460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.510634 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.510615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-telemeter-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.510770 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.510754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.510931 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.510913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/79d2fdb6-f33f-435e-93aa-81785774bdd8-federate-client-tls\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.515807 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.515788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf79\" (UniqueName: \"kubernetes.io/projected/79d2fdb6-f33f-435e-93aa-81785774bdd8-kube-api-access-8wf79\") pod \"telemeter-client-797975fdf9-7lnv6\" (UID: \"79d2fdb6-f33f-435e-93aa-81785774bdd8\") " pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.714180 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.714099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6"
Sep 04 22:49:43.987008 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:43.986924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-797975fdf9-7lnv6"]
Sep 04 22:49:43.992229 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:43.992187    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d2fdb6_f33f_435e_93aa_81785774bdd8.slice/crio-7d7ec79fe769d6be200cb351aed0c7bb293731472cccfa96cfac42a10595a42b WatchSource:0}: Error finding container 7d7ec79fe769d6be200cb351aed0c7bb293731472cccfa96cfac42a10595a42b: Status 404 returned error can't find the container with id 7d7ec79fe769d6be200cb351aed0c7bb293731472cccfa96cfac42a10595a42b
Sep 04 22:49:44.313105 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:44.313009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6" event={"ID":"79d2fdb6-f33f-435e-93aa-81785774bdd8","Type":"ContainerStarted","Data":"7d7ec79fe769d6be200cb351aed0c7bb293731472cccfa96cfac42a10595a42b"}
Sep 04 22:49:45.247951 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:45.247910    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:49:45.248375 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:45.247965    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:49:46.323359 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:46.323278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6" event={"ID":"79d2fdb6-f33f-435e-93aa-81785774bdd8","Type":"ContainerStarted","Data":"b4103e224384fe9867321a5c1df3a71770ab982f1830dacdb2e84f6993eeb51d"}
Sep 04 22:49:47.327263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.327182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6" event={"ID":"79d2fdb6-f33f-435e-93aa-81785774bdd8","Type":"ContainerStarted","Data":"5dfe1766cb2868b851bdfaaead158aad211bd71dfb56209d77292d11b5fcc546"}
Sep 04 22:49:47.327631 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.327285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6" event={"ID":"79d2fdb6-f33f-435e-93aa-81785774bdd8","Type":"ContainerStarted","Data":"e0aa15e37c8d6a4b9d6bf6d51eb4841591d13b0552832bcc6fc164997e3bcb62"}
Sep 04 22:49:47.349423 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.349359    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-797975fdf9-7lnv6" podStartSLOduration=1.542523514 podStartE2EDuration="4.349341019s" totalImagesPullingTime="2.806817505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:49:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:47.348947193 +0000 UTC m=+271.520021806" watchObservedRunningTime="2026-09-04 22:49:47.349341019 +0000 UTC m=+271.520415633"
Sep 04 22:49:47.508547 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.508506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:49:47.511721 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.511698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.514385 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.514364    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 04 22:49:47.514625 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.514607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 04 22:49:47.514713 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.514640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 04 22:49:47.514811 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.514792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 04 22:49:47.515035 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.515014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nrl48\""
Sep 04 22:49:47.515100 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.515039    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 04 22:49:47.515156 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.515113    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 04 22:49:47.515550 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.515528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 04 22:49:47.519616 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.519600    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 04 22:49:47.525221 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.525194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:49:47.542232 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9d4\" (UniqueName: \"kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542351 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.542639 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.542448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.643819 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9d4\" (UniqueName: \"kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644003 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.643971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644248 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.644013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644611 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.644582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644704 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.644678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644792 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.644756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.644792 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.644775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.646311 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.646287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.646486 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.646468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.652412 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.652391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9d4\" (UniqueName: \"kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4\") pod \"console-574c95b88f-dbmdd\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") " pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:47.821456 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:47.821414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:48.091207 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:48.091172    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:49:48.096514 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:49:48.096484    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c38288_c3ea_40ad_9938_56e1586c54e4.slice/crio-488b17982ef1aa89c3d73e1793e9273de3b227749fba3b54b5ef84de37506997 WatchSource:0}: Error finding container 488b17982ef1aa89c3d73e1793e9273de3b227749fba3b54b5ef84de37506997: Status 404 returned error can't find the container with id 488b17982ef1aa89c3d73e1793e9273de3b227749fba3b54b5ef84de37506997
Sep 04 22:49:48.331428 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:48.331394    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c95b88f-dbmdd" event={"ID":"91c38288-c3ea-40ad-9938-56e1586c54e4","Type":"ContainerStarted","Data":"488b17982ef1aa89c3d73e1793e9273de3b227749fba3b54b5ef84de37506997"}
Sep 04 22:49:51.342943 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:51.342905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c95b88f-dbmdd" event={"ID":"91c38288-c3ea-40ad-9938-56e1586c54e4","Type":"ContainerStarted","Data":"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"}
Sep 04 22:49:51.360590 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:51.360533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-574c95b88f-dbmdd" podStartSLOduration=1.76873466 podStartE2EDuration="4.360515413s" totalImagesPullingTime="2.591780753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:49:51.359301789 +0000 UTC m=+275.530376400" watchObservedRunningTime="2026-09-04 22:49:51.360515413 +0000 UTC m=+275.531590053"
Sep 04 22:49:52.401576 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:52.401536    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" podUID="4924cb65-f77e-437e-832e-fb749d30e111" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 04 22:49:53.344386 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:53.344350    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:49:53.344579 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:53.344402    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:49:55.247135 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:55.247100    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:49:55.247516 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:55.247155    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:49:56.965137 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:49:56.965084    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6pmhs" podUID="217286bb-2071-42aa-b834-bdf9a1b3ad85"
Sep 04 22:49:57.363047 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:57.363014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:49:57.821993 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:57.821890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:57.821993 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:57.821973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:49:57.823229 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:57.823206    2577 patch_prober.go:28] interesting pod/console-574c95b88f-dbmdd container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.13:8443/health\": dial tcp 10.134.0.13:8443: connect: connection refused" start-of-body=
Sep 04 22:49:57.823311 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:49:57.823248    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-574c95b88f-dbmdd" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerName="console" probeResult="failure" output="Get \"https://10.134.0.13:8443/health\": dial tcp 10.134.0.13:8443: connect: connection refused"
Sep 04 22:50:00.644487 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:00.644445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:50:00.646780 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:00.646750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/217286bb-2071-42aa-b834-bdf9a1b3ad85-metrics-tls\") pod \"dns-default-6pmhs\" (UID: \"217286bb-2071-42aa-b834-bdf9a1b3ad85\") " pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:50:00.666255 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:00.666223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cv6t6\""
Sep 04 22:50:00.674121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:00.674093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:50:00.945095 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:00.945064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pmhs"]
Sep 04 22:50:00.947453 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:50:00.947423    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217286bb_2071_42aa_b834_bdf9a1b3ad85.slice/crio-90d03845c57e748fc1d63edffaf9ab9d17f14b3f3335eaf37c059d133fba82b3 WatchSource:0}: Error finding container 90d03845c57e748fc1d63edffaf9ab9d17f14b3f3335eaf37c059d133fba82b3: Status 404 returned error can't find the container with id 90d03845c57e748fc1d63edffaf9ab9d17f14b3f3335eaf37c059d133fba82b3
Sep 04 22:50:01.377197 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:01.377160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pmhs" event={"ID":"217286bb-2071-42aa-b834-bdf9a1b3ad85","Type":"ContainerStarted","Data":"90d03845c57e748fc1d63edffaf9ab9d17f14b3f3335eaf37c059d133fba82b3"}
Sep 04 22:50:02.401347 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:02.401304    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" podUID="4924cb65-f77e-437e-832e-fb749d30e111" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 04 22:50:02.401780 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:02.401387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg"
Sep 04 22:50:02.402033 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:02.401996    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c546c87cdce43c23697f1f738e84056022ac52ea35a9fc25ea64d4fc68e4927f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 04 22:50:02.402086 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:02.402057    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" podUID="4924cb65-f77e-437e-832e-fb749d30e111" containerName="service-proxy" containerID="cri-o://c546c87cdce43c23697f1f738e84056022ac52ea35a9fc25ea64d4fc68e4927f" gracePeriod=30
Sep 04 22:50:03.344624 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.344582    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:50:03.344813 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.344641    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:50:03.344813 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.344690    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:50:03.345260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.345233    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"31f8077f1a7f841cf9524c6292efc21febdab7e6e3a3518073d084c9257c5bed"} pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 04 22:50:03.348608 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.348584    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:50:03.348699 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.348628    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:50:03.387508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.387476    2577 generic.go:356] "Generic (PLEG): container finished" podID="4924cb65-f77e-437e-832e-fb749d30e111" containerID="c546c87cdce43c23697f1f738e84056022ac52ea35a9fc25ea64d4fc68e4927f" exitCode=2
Sep 04 22:50:03.387683 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.387554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerDied","Data":"c546c87cdce43c23697f1f738e84056022ac52ea35a9fc25ea64d4fc68e4927f"}
Sep 04 22:50:03.387683 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.387602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-845bc7bcb4-gfvtg" event={"ID":"4924cb65-f77e-437e-832e-fb749d30e111","Type":"ContainerStarted","Data":"0b7a9b13d734fba72b519fef9ba80e322c77ee890e0fdf55414b4ef27bc06e6a"}
Sep 04 22:50:03.389263 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.389241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pmhs" event={"ID":"217286bb-2071-42aa-b834-bdf9a1b3ad85","Type":"ContainerStarted","Data":"d28b2f19256d72b4d7b172977bac534dd5b5165606bf07cbcd69d3468ea528e9"}
Sep 04 22:50:03.389362 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.389287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:50:03.389362 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.389303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pmhs" event={"ID":"217286bb-2071-42aa-b834-bdf9a1b3ad85","Type":"ContainerStarted","Data":"81db60dfd312858de9c1f1803f6d7ffa1800438725298cbde82bc768b550374a"}
Sep 04 22:50:03.420560 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:03.420512    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6pmhs" podStartSLOduration=251.423816623 podStartE2EDuration="4m13.420494399s" totalImagesPullingTime="1.996677776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:50:03.419239354 +0000 UTC m=+287.590313963" watchObservedRunningTime="2026-09-04 22:50:03.420494399 +0000 UTC m=+287.591569015"
Sep 04 22:50:07.826163 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:07.826126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:50:07.829948 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:07.829925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:50:09.915826 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:09.915797    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pmhs_217286bb-2071-42aa-b834-bdf9a1b3ad85/dns/0.log"
Sep 04 22:50:10.116324 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:10.116287    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pmhs_217286bb-2071-42aa-b834-bdf9a1b3ad85/kube-rbac-proxy/0.log"
Sep 04 22:50:11.515950 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:11.515923    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhj5d_de5fdc91-819b-4568-bb01-54547f4735cd/dns-node-resolver/0.log"
Sep 04 22:50:11.916034 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:11.916006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c9975d4d-dgwj2_50fcc1b1-f4ef-4b49-a24d-6b18a3318353/registry/0.log"
Sep 04 22:50:12.315794 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:12.315703    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m424m_4ee6f1db-453b-4985-aa29-774fd08ba876/node-ca/0.log"
Sep 04 22:50:13.116048 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:13.116018    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjcjb_b0618c88-28ff-4047-b6a4-36a3088c7e49/serve-healthcheck-canary/0.log"
Sep 04 22:50:13.348858 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:13.348826    2577 patch_prober.go:28] interesting pod/image-registry-85c9975d4d-dgwj2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 04 22:50:13.349024 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:13.348876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:50:13.395341 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:13.395260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6pmhs"
Sep 04 22:50:16.215656 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:16.215624    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 04 22:50:23.349513 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:23.349481    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:50:28.364091 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:28.364040    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" podUID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerName="registry" containerID="cri-o://31f8077f1a7f841cf9524c6292efc21febdab7e6e3a3518073d084c9257c5bed" gracePeriod=30
Sep 04 22:50:29.479375 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:29.479352    2577 generic.go:356] "Generic (PLEG): container finished" podID="50fcc1b1-f4ef-4b49-a24d-6b18a3318353" containerID="31f8077f1a7f841cf9524c6292efc21febdab7e6e3a3518073d084c9257c5bed" exitCode=0
Sep 04 22:50:29.479654 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:29.479421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" event={"ID":"50fcc1b1-f4ef-4b49-a24d-6b18a3318353","Type":"ContainerDied","Data":"31f8077f1a7f841cf9524c6292efc21febdab7e6e3a3518073d084c9257c5bed"}
Sep 04 22:50:30.485078 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:30.485031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2" event={"ID":"50fcc1b1-f4ef-4b49-a24d-6b18a3318353","Type":"ContainerStarted","Data":"96b2d4e088689abe80e4f08956d3f48b94874b1d63d529c66637fb09c5c138d2"}
Sep 04 22:50:30.485519 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:30.485104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:50:51.493662 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:50:51.493630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85c9975d4d-dgwj2"
Sep 04 22:51:09.721641 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:09.721568    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:51:34.740487 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:34.740418    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-574c95b88f-dbmdd" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerName="console" containerID="cri-o://08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5" gracePeriod=15
Sep 04 22:51:34.974608 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:34.974585    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:51:35.044275 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l9d4\" (UniqueName: \"kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044275 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044521 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044284    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044521 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044320    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044521 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044521 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044521 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config\") pod \"91c38288-c3ea-40ad-9938-56e1586c54e4\" (UID: \"91c38288-c3ea-40ad-9938-56e1586c54e4\") "
Sep 04 22:51:35.044827 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:51:35.044888 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:51:35.044888 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca" (OuterVolumeSpecName: "service-ca") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:51:35.044888 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.044866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config" (OuterVolumeSpecName: "console-config") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:51:35.046728 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.046700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:51:35.046934 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.046914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4" (OuterVolumeSpecName: "kube-api-access-6l9d4") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "kube-api-access-6l9d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:51:35.046934 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.046922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91c38288-c3ea-40ad-9938-56e1586c54e4" (UID: "91c38288-c3ea-40ad-9938-56e1586c54e4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:51:35.145575 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145535    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-service-ca\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145575 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145569    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-serving-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145575 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145581    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-oauth-serving-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145850 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145590    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91c38288-c3ea-40ad-9938-56e1586c54e4-console-oauth-config\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145850 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145600    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-console-config\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145850 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145608    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l9d4\" (UniqueName: \"kubernetes.io/projected/91c38288-c3ea-40ad-9938-56e1586c54e4-kube-api-access-6l9d4\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.145850 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.145616    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91c38288-c3ea-40ad-9938-56e1586c54e4-trusted-ca-bundle\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:51:35.706841 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.706804    2577 generic.go:356] "Generic (PLEG): container finished" podID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerID="08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5" exitCode=0
Sep 04 22:51:35.707010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.706857    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c95b88f-dbmdd"
Sep 04 22:51:35.707010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.706864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c95b88f-dbmdd" event={"ID":"91c38288-c3ea-40ad-9938-56e1586c54e4","Type":"ContainerDied","Data":"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"}
Sep 04 22:51:35.707010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.706934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c95b88f-dbmdd" event={"ID":"91c38288-c3ea-40ad-9938-56e1586c54e4","Type":"ContainerDied","Data":"488b17982ef1aa89c3d73e1793e9273de3b227749fba3b54b5ef84de37506997"}
Sep 04 22:51:35.707010 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.706955    2577 scope.go:117] "RemoveContainer" containerID="08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"
Sep 04 22:51:35.716117 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.716098    2577 scope.go:117] "RemoveContainer" containerID="08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"
Sep 04 22:51:35.716381 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:51:35.716363    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5\": container with ID starting with 08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5 not found: ID does not exist" containerID="08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"
Sep 04 22:51:35.716444 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.716386    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5"} err="failed to get container status \"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5\": rpc error: code = NotFound desc = could not find container \"08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5\": container with ID starting with 08172021a54c3b0192a2a130db22598e9550594663d4307d262283f81f51b2f5 not found: ID does not exist"
Sep 04 22:51:35.727953 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.727929    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:51:35.737918 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:35.737895    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574c95b88f-dbmdd"]
Sep 04 22:51:36.324025 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:51:36.323993    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" path="/var/lib/kubelet/pods/91c38288-c3ea-40ad-9938-56e1586c54e4/volumes"
Sep 04 22:53:10.600024 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.599983    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"]
Sep 04 22:53:10.600591 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.600326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerName="console"
Sep 04 22:53:10.600591 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.600342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerName="console"
Sep 04 22:53:10.600591 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.600397    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c38288-c3ea-40ad-9938-56e1586c54e4" containerName="console"
Sep 04 22:53:10.603774 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.603752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.606310 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.606288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 04 22:53:10.606449 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.606326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d78dr\""
Sep 04 22:53:10.606523 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.606450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 04 22:53:10.606627 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.606611    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 04 22:53:10.606715 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.606615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 04 22:53:10.620671 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.620641    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"]
Sep 04 22:53:10.648914 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.648877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.648914 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.648921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-webhook-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.649132 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.648952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5v4k\" (UniqueName: \"kubernetes.io/projected/fd98bc56-04c4-42ed-ba22-140583a8f06c-kube-api-access-c5v4k\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.649132 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.649022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-manifests\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.649132 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.649065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-tmp\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750061 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750061 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-webhook-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750347 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5v4k\" (UniqueName: \"kubernetes.io/projected/fd98bc56-04c4-42ed-ba22-140583a8f06c-kube-api-access-c5v4k\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750347 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-manifests\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750347 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-tmp\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.750677 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.750647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-manifests\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.752463 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.752438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd98bc56-04c4-42ed-ba22-140583a8f06c-tmp\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.752720 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.752700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-webhook-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.752813 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.752758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd98bc56-04c4-42ed-ba22-140583a8f06c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.761154 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.761124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5v4k\" (UniqueName: \"kubernetes.io/projected/fd98bc56-04c4-42ed-ba22-140583a8f06c-kube-api-access-c5v4k\") pod \"opendatahub-operator-controller-manager-759c5d6957-xf8ps\" (UID: \"fd98bc56-04c4-42ed-ba22-140583a8f06c\") " pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:10.913466 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:10.913370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:11.208552 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:11.208463    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"]
Sep 04 22:53:11.212068 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:53:11.212034    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd98bc56_04c4_42ed_ba22_140583a8f06c.slice/crio-026a23c58de701666ab938c030a09b63950b5bc6b0e34f3cc9dc0cea068ce2a9 WatchSource:0}: Error finding container 026a23c58de701666ab938c030a09b63950b5bc6b0e34f3cc9dc0cea068ce2a9: Status 404 returned error can't find the container with id 026a23c58de701666ab938c030a09b63950b5bc6b0e34f3cc9dc0cea068ce2a9
Sep 04 22:53:11.213671 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:11.213656    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:53:12.032461 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:12.032422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps" event={"ID":"fd98bc56-04c4-42ed-ba22-140583a8f06c","Type":"ContainerStarted","Data":"026a23c58de701666ab938c030a09b63950b5bc6b0e34f3cc9dc0cea068ce2a9"}
Sep 04 22:53:15.045888 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:15.045849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps" event={"ID":"fd98bc56-04c4-42ed-ba22-140583a8f06c","Type":"ContainerStarted","Data":"6638ea44156a692aca20e4fe95436f099ff49217120d5bb026d8aabe74fe5c66"}
Sep 04 22:53:16.050592 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:16.050552    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd98bc56-04c4-42ed-ba22-140583a8f06c" containerID="6638ea44156a692aca20e4fe95436f099ff49217120d5bb026d8aabe74fe5c66" exitCode=0
Sep 04 22:53:16.051025 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:16.050638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps" event={"ID":"fd98bc56-04c4-42ed-ba22-140583a8f06c","Type":"ContainerDied","Data":"6638ea44156a692aca20e4fe95436f099ff49217120d5bb026d8aabe74fe5c66"}
Sep 04 22:53:17.056288 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:17.056245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps" event={"ID":"fd98bc56-04c4-42ed-ba22-140583a8f06c","Type":"ContainerStarted","Data":"ca40c92ff472b59e364c77c85b56205cea40a5070d4247cb30335fc56f71fc89"}
Sep 04 22:53:17.056797 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:17.056327    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:17.080006 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:17.079947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps" podStartSLOduration=2.707087525 podStartE2EDuration="7.079930069s" totalImagesPullingTime="3.33161733s" totalInitContainerRuntime="1.041225214s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:53:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:53:17.078230424 +0000 UTC m=+481.249305036" watchObservedRunningTime="2026-09-04 22:53:17.079930069 +0000 UTC m=+481.251004680"
Sep 04 22:53:28.062504 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:28.062472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-759c5d6957-xf8ps"
Sep 04 22:53:30.203849 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.203816    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"]
Sep 04 22:53:30.214413 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.213955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.216678 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.216656    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 04 22:53:30.216787 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.216655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 04 22:53:30.217180 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.217161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 04 22:53:30.217458 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.217195    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 04 22:53:30.221312 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.221283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"]
Sep 04 22:53:30.301577 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.301537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ct2\" (UniqueName: \"kubernetes.io/projected/1c7b3810-93fb-4407-b305-b5651a1d78f3-kube-api-access-99ct2\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.301577 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.301582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7b3810-93fb-4407-b305-b5651a1d78f3-tmp\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.301820 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.301611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.402623 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.402581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.402840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.402656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ct2\" (UniqueName: \"kubernetes.io/projected/1c7b3810-93fb-4407-b305-b5651a1d78f3-kube-api-access-99ct2\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.402840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.402679    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7b3810-93fb-4407-b305-b5651a1d78f3-tmp\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.402840 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:53:30.402769    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 04 22:53:30.402995 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:53:30.402849    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs podName:1c7b3810-93fb-4407-b305-b5651a1d78f3 nodeName:}" failed. No retries permitted until 2026-09-04 22:53:30.902824619 +0000 UTC m=+495.073899227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs") pod "kube-auth-proxy-c4db4689d-r4bvs" (UID: "1c7b3810-93fb-4407-b305-b5651a1d78f3") : secret "kube-auth-proxy-tls" not found
Sep 04 22:53:30.404958 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.404923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c7b3810-93fb-4407-b305-b5651a1d78f3-tmp\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.413437 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.413410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ct2\" (UniqueName: \"kubernetes.io/projected/1c7b3810-93fb-4407-b305-b5651a1d78f3-kube-api-access-99ct2\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.906962 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.906923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:30.909264 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:30.909243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c7b3810-93fb-4407-b305-b5651a1d78f3-tls-certs\") pod \"kube-auth-proxy-c4db4689d-r4bvs\" (UID: \"1c7b3810-93fb-4407-b305-b5651a1d78f3\") " pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:31.128161 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.128121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"
Sep 04 22:53:31.403038 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.403008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs"]
Sep 04 22:53:31.410167 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:53:31.410127    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7b3810_93fb_4407_b305_b5651a1d78f3.slice/crio-038a8fe18f1580b86c8750c778033b573f50908574be758380df158571664c7f WatchSource:0}: Error finding container 038a8fe18f1580b86c8750c778033b573f50908574be758380df158571664c7f: Status 404 returned error can't find the container with id 038a8fe18f1580b86c8750c778033b573f50908574be758380df158571664c7f
Sep 04 22:53:31.483816 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.483721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:31.486554 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.486539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.489498 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.489461    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 22:53:31.489715 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.489694    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 04 22:53:31.489715 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.489712    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-n5s97\""
Sep 04 22:53:31.489992 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.489975    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 04 22:53:31.490218 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.490204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 04 22:53:31.501793 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.501766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:31.610869 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.610822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.610869 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.610878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.611121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.610955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.611121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.610995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.611121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.611017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.611121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.611074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.611121 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.611103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpnv\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711461 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711605 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711605 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711694 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711730 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711818 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpnv\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.711818 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.711809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.712075 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.712051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.713840 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.713817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.714032 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.714013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.714094 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.714046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.714094 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.714070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.720034 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.720014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpnv\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.720141 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.720122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9xxg2\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:31.796886 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:31.796790    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:32.072680 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:32.072602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:32.075550 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:53:32.075522    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef4ed92_cbb2_4479_9f3f_3167a1b48ecd.slice/crio-afca93d8f42cd30411c4b93975da6250adabe81ea4a33a2755933573125321a3 WatchSource:0}: Error finding container afca93d8f42cd30411c4b93975da6250adabe81ea4a33a2755933573125321a3: Status 404 returned error can't find the container with id afca93d8f42cd30411c4b93975da6250adabe81ea4a33a2755933573125321a3
Sep 04 22:53:32.106074 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:32.106040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs" event={"ID":"1c7b3810-93fb-4407-b305-b5651a1d78f3","Type":"ContainerStarted","Data":"038a8fe18f1580b86c8750c778033b573f50908574be758380df158571664c7f"}
Sep 04 22:53:32.107196 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:32.107177    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" event={"ID":"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd","Type":"ContainerStarted","Data":"afca93d8f42cd30411c4b93975da6250adabe81ea4a33a2755933573125321a3"}
Sep 04 22:53:35.751397 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:35.751360    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 22:53:35.751782 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:35.751447    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 22:53:36.125213 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.125178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" event={"ID":"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd","Type":"ContainerStarted","Data":"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"}
Sep 04 22:53:36.125427 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.125380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:36.126967 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.126936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs" event={"ID":"1c7b3810-93fb-4407-b305-b5651a1d78f3","Type":"ContainerStarted","Data":"fff5f76d4961944970121dabf8c4d0af69b49c4d8b85a61dab74fa3768b4799e"}
Sep 04 22:53:36.127332 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.127310    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9xxg2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:53:36.127445 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.127349    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:53:36.144196 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.144154    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" podStartSLOduration=1.4706908269999999 podStartE2EDuration="5.144138884s" totalImagesPullingTime="3.673448057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:53:36.143132084 +0000 UTC m=+500.314206695" watchObservedRunningTime="2026-09-04 22:53:36.144138884 +0000 UTC m=+500.315213486"
Sep 04 22:53:36.158932 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:36.158885    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c4db4689d-r4bvs" podStartSLOduration=1.866006895 podStartE2EDuration="6.15886414s" totalImagesPullingTime="4.292857245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:53:36.158080385 +0000 UTC m=+500.329155024" watchObservedRunningTime="2026-09-04 22:53:36.15886414 +0000 UTC m=+500.329938772"
Sep 04 22:53:37.131335 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:37.131303    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-9xxg2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:53:37.131674 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:37.131353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:53:40.132226 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:40.132192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:48.236543 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.236502    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"]
Sep 04 22:53:48.239731 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.239709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.243615 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.243587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 04 22:53:48.243772 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.243594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 04 22:53:48.244193 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.243980    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 04 22:53:48.244531 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.244369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 04 22:53:48.244531 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.244386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:53:48.244531 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.244268    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-p2snh\""
Sep 04 22:53:48.251375 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.251349    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"]
Sep 04 22:53:48.340340 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.340303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d0628fa-a12f-4748-9f29-fa9052233c33-manager-config\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.340340 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.340343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hhc2\" (UniqueName: \"kubernetes.io/projected/3d0628fa-a12f-4748-9f29-fa9052233c33-kube-api-access-2hhc2\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.340548 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.340366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.340548 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.340437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-metrics-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.441634 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.441602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hhc2\" (UniqueName: \"kubernetes.io/projected/3d0628fa-a12f-4748-9f29-fa9052233c33-kube-api-access-2hhc2\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.441826 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.441643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.441826 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.441691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-metrics-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.441826 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.441764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d0628fa-a12f-4748-9f29-fa9052233c33-manager-config\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.442467 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.442447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3d0628fa-a12f-4748-9f29-fa9052233c33-manager-config\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.444194 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.444164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-metrics-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.444194 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.444187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d0628fa-a12f-4748-9f29-fa9052233c33-cert\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.452690 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.452656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hhc2\" (UniqueName: \"kubernetes.io/projected/3d0628fa-a12f-4748-9f29-fa9052233c33-kube-api-access-2hhc2\") pod \"lws-controller-manager-6c6bd7c7b8-zcgdg\" (UID: \"3d0628fa-a12f-4748-9f29-fa9052233c33\") " pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.554196 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.554101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:53:48.824791 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:48.824705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"]
Sep 04 22:53:48.828925 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:53:48.828897    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0628fa_a12f_4748_9f29_fa9052233c33.slice/crio-1755bc241f3cd587d8a85170adcc28103a762dcbf42b3506b90d7d5ced6bb4ce WatchSource:0}: Error finding container 1755bc241f3cd587d8a85170adcc28103a762dcbf42b3506b90d7d5ced6bb4ce: Status 404 returned error can't find the container with id 1755bc241f3cd587d8a85170adcc28103a762dcbf42b3506b90d7d5ced6bb4ce
Sep 04 22:53:49.172803 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:49.172765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg" event={"ID":"3d0628fa-a12f-4748-9f29-fa9052233c33","Type":"ContainerStarted","Data":"1755bc241f3cd587d8a85170adcc28103a762dcbf42b3506b90d7d5ced6bb4ce"}
Sep 04 22:53:52.476295 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.476261    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:52.476753 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.476448    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery" containerID="cri-o://95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a" gracePeriod=30
Sep 04 22:53:52.715933 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.715906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:52.876433 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lpnv\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876557    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876635 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig\") pod \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\" (UID: \"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd\") "
Sep 04 22:53:52.876978 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.876946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:53:52.879038 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.878985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:53:52.879038 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.879016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:53:52.879280 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.879256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv" (OuterVolumeSpecName: "kube-api-access-5lpnv") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "kube-api-access-5lpnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:53:52.879352 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.879305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts" (OuterVolumeSpecName: "cacerts") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:53:52.879352 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.879328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token" (OuterVolumeSpecName: "istio-token") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:53:52.879422 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.879371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs" (OuterVolumeSpecName: "local-certs") pod "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" (UID: "eef4ed92-cbb2-4479-9f3f-3167a1b48ecd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:53:52.977954 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.977918    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-ca-configmap\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.977969    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-local-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.977982    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-cacerts\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.977991    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.977999    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lpnv\" (UniqueName: \"kubernetes.io/projected/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-kube-api-access-5lpnv\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.978007    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-csr-dns-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:52.978120 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:52.978015    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd-istio-kubeconfig\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:53:53.188877 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.188791    2577 generic.go:356] "Generic (PLEG): container finished" podID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerID="95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a" exitCode=0
Sep 04 22:53:53.188877 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.188842    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"
Sep 04 22:53:53.189049 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.188840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" event={"ID":"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd","Type":"ContainerDied","Data":"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"}
Sep 04 22:53:53.189049 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.188972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2" event={"ID":"eef4ed92-cbb2-4479-9f3f-3167a1b48ecd","Type":"ContainerDied","Data":"afca93d8f42cd30411c4b93975da6250adabe81ea4a33a2755933573125321a3"}
Sep 04 22:53:53.189049 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.189002    2577 scope.go:117] "RemoveContainer" containerID="95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"
Sep 04 22:53:53.197903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.197883    2577 scope.go:117] "RemoveContainer" containerID="95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"
Sep 04 22:53:53.198207 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:53:53.198186    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a\": container with ID starting with 95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a not found: ID does not exist" containerID="95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"
Sep 04 22:53:53.198262 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.198215    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a"} err="failed to get container status \"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a\": rpc error: code = NotFound desc = could not find container \"95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a\": container with ID starting with 95f9d453116b79fe44bc7ffd176b5cb9272d774c7915f685a4659924e588975a not found: ID does not exist"
Sep 04 22:53:53.225210 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.225174    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:53.230404 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:53.230377    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9xxg2"]
Sep 04 22:53:54.324165 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:53:54.324131    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" path="/var/lib/kubelet/pods/eef4ed92-cbb2-4479-9f3f-3167a1b48ecd/volumes"
Sep 04 22:54:03.226516 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:54:03.226477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg" event={"ID":"3d0628fa-a12f-4748-9f29-fa9052233c33","Type":"ContainerStarted","Data":"eb41d43acbc72b254490fa2bafc09640f56164d71a82bc5794fd89be1f02cb1e"}
Sep 04 22:54:03.226930 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:54:03.226608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:54:03.241915 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:54:03.241862    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg" podStartSLOduration=1.752624318 podStartE2EDuration="15.241847221s" totalImagesPullingTime="13.489222903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:54:03.241099289 +0000 UTC m=+527.412173901" watchObservedRunningTime="2026-09-04 22:54:03.241847221 +0000 UTC m=+527.412921833"
Sep 04 22:54:14.233043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:54:14.232967    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c6bd7c7b8-zcgdg"
Sep 04 22:55:14.147200 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.147160    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"]
Sep 04 22:55:14.147609 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.147422    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery"
Sep 04 22:55:14.147609 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.147433    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery"
Sep 04 22:55:14.147609 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.147489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef4ed92-cbb2-4479-9f3f-3167a1b48ecd" containerName="discovery"
Sep 04 22:55:14.150250 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.150233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.152572 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.152546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 04 22:55:14.152713 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.152578    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 04 22:55:14.152713 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.152579    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 04 22:55:14.153370 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.153351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9x9r4\""
Sep 04 22:55:14.153443 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.153351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 04 22:55:14.161311 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.161284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"]
Sep 04 22:55:14.189285 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.189249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fngj\" (UniqueName: \"kubernetes.io/projected/de8bf063-0a96-49e6-a812-665f62ce36c6-kube-api-access-8fngj\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.189448 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.189294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de8bf063-0a96-49e6-a812-665f62ce36c6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.189448 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.189359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.289704 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.289669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de8bf063-0a96-49e6-a812-665f62ce36c6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.289900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.289720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.289900 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.289775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fngj\" (UniqueName: \"kubernetes.io/projected/de8bf063-0a96-49e6-a812-665f62ce36c6-kube-api-access-8fngj\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.289972 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:55:14.289894    2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 04 22:55:14.289972 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:55:14.289968    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert podName:de8bf063-0a96-49e6-a812-665f62ce36c6 nodeName:}" failed. No retries permitted until 2026-09-04 22:55:14.789947 +0000 UTC m=+598.961021593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-n9rx4" (UID: "de8bf063-0a96-49e6-a812-665f62ce36c6") : secret "plugin-serving-cert" not found
Sep 04 22:55:14.290324 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.290305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/de8bf063-0a96-49e6-a812-665f62ce36c6-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.304904 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.304869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fngj\" (UniqueName: \"kubernetes.io/projected/de8bf063-0a96-49e6-a812-665f62ce36c6-kube-api-access-8fngj\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.793645 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.793608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:14.796097 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:14.796071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8bf063-0a96-49e6-a812-665f62ce36c6-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-n9rx4\" (UID: \"de8bf063-0a96-49e6-a812-665f62ce36c6\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:15.059205 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:15.059115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"
Sep 04 22:55:15.330054 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:15.330031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4"]
Sep 04 22:55:15.332613 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:55:15.332587    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde8bf063_0a96_49e6_a812_665f62ce36c6.slice/crio-c66f9c4a0af0d59ee4ae54be873ab3f6e71ab1975e1052ffb2cfe698e098c462 WatchSource:0}: Error finding container c66f9c4a0af0d59ee4ae54be873ab3f6e71ab1975e1052ffb2cfe698e098c462: Status 404 returned error can't find the container with id c66f9c4a0af0d59ee4ae54be873ab3f6e71ab1975e1052ffb2cfe698e098c462
Sep 04 22:55:15.489778 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:15.489718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4" event={"ID":"de8bf063-0a96-49e6-a812-665f62ce36c6","Type":"ContainerStarted","Data":"c66f9c4a0af0d59ee4ae54be873ab3f6e71ab1975e1052ffb2cfe698e098c462"}
Sep 04 22:55:20.510637 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:20.510599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4" event={"ID":"de8bf063-0a96-49e6-a812-665f62ce36c6","Type":"ContainerStarted","Data":"73e64c8650849e68512aff29fd0c43a09f4ed73323549c3ab935824e69d2cefd"}
Sep 04 22:55:20.527289 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:55:20.527244    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-n9rx4" podStartSLOduration=2.087002805 podStartE2EDuration="6.527228449s" totalImagesPullingTime="4.440225644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:55:20.525512241 +0000 UTC m=+604.696586855" watchObservedRunningTime="2026-09-04 22:55:20.527228449 +0000 UTC m=+604.698303060"
Sep 04 22:56:09.806586 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.806549    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:56:09.809711 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.809694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:09.812175 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.812142    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 04 22:56:09.820052 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.820030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:56:09.851329 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.851294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:56:09.916112 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.916065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7778dbc0-aa1d-4071-bcb1-baec439b937a-config-file\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:09.916277 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:09.916176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5qh\" (UniqueName: \"kubernetes.io/projected/7778dbc0-aa1d-4071-bcb1-baec439b937a-kube-api-access-qx5qh\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.016754 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.016707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5qh\" (UniqueName: \"kubernetes.io/projected/7778dbc0-aa1d-4071-bcb1-baec439b937a-kube-api-access-qx5qh\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.016918 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.016800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7778dbc0-aa1d-4071-bcb1-baec439b937a-config-file\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.017442 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.017423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7778dbc0-aa1d-4071-bcb1-baec439b937a-config-file\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.025343 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.025319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5qh\" (UniqueName: \"kubernetes.io/projected/7778dbc0-aa1d-4071-bcb1-baec439b937a-kube-api-access-qx5qh\") pod \"limitador-limitador-85d9db4d5d-cwbt6\" (UID: \"7778dbc0-aa1d-4071-bcb1-baec439b937a\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.119502 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.119468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:10.294234 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.294198    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:10.297416 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.297402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:10.299908 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.299889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fhw8g\""
Sep 04 22:56:10.304883 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.304863    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:10.389765 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.389586    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:56:10.392491 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:56:10.392461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7778dbc0_aa1d_4071_bcb1_baec439b937a.slice/crio-ebf5f68b7694ce32a80d9f9169f39a731fc42df0efcaf7f31849d51aebd7a017 WatchSource:0}: Error finding container ebf5f68b7694ce32a80d9f9169f39a731fc42df0efcaf7f31849d51aebd7a017: Status 404 returned error can't find the container with id ebf5f68b7694ce32a80d9f9169f39a731fc42df0efcaf7f31849d51aebd7a017
Sep 04 22:56:10.413860 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.413832    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:10.416711 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.416696    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:10.419004 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.418979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c54\" (UniqueName: \"kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54\") pod \"authorino-b54df94b4-nqzzn\" (UID: \"007ed975-6abd-4255-9a63-f450245675f1\") " pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:10.424382 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.424362    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:10.519825 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.519792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c54\" (UniqueName: \"kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54\") pod \"authorino-b54df94b4-nqzzn\" (UID: \"007ed975-6abd-4255-9a63-f450245675f1\") " pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:10.520000 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.519837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xl8h\" (UniqueName: \"kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h\") pod \"authorino-c548b5d7d-mqggf\" (UID: \"65cf4b75-4292-41a2-a3f0-4b107a3876be\") " pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:10.528692 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.528669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c54\" (UniqueName: \"kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54\") pod \"authorino-b54df94b4-nqzzn\" (UID: \"007ed975-6abd-4255-9a63-f450245675f1\") " pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:10.607303 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.607262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:10.620526 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.620500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xl8h\" (UniqueName: \"kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h\") pod \"authorino-c548b5d7d-mqggf\" (UID: \"65cf4b75-4292-41a2-a3f0-4b107a3876be\") " pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:10.631160 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.631132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xl8h\" (UniqueName: \"kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h\") pod \"authorino-c548b5d7d-mqggf\" (UID: \"65cf4b75-4292-41a2-a3f0-4b107a3876be\") " pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:10.689043 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.689006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6" event={"ID":"7778dbc0-aa1d-4071-bcb1-baec439b937a","Type":"ContainerStarted","Data":"ebf5f68b7694ce32a80d9f9169f39a731fc42df0efcaf7f31849d51aebd7a017"}
Sep 04 22:56:10.725311 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.725278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:10.874565 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.874526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:10.877084 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:56:10.877062    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod007ed975_6abd_4255_9a63_f450245675f1.slice/crio-df6adae47823f09b88ab6926a9d63757501e35c8bca19d7d40f909177b8f5eea WatchSource:0}: Error finding container df6adae47823f09b88ab6926a9d63757501e35c8bca19d7d40f909177b8f5eea: Status 404 returned error can't find the container with id df6adae47823f09b88ab6926a9d63757501e35c8bca19d7d40f909177b8f5eea
Sep 04 22:56:10.993349 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:10.993321    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:10.995273 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:56:10.995246    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65cf4b75_4292_41a2_a3f0_4b107a3876be.slice/crio-f3241b76dda919957e6293d76ac1bbe4e67d220e69768102af1c42d312df0a4f WatchSource:0}: Error finding container f3241b76dda919957e6293d76ac1bbe4e67d220e69768102af1c42d312df0a4f: Status 404 returned error can't find the container with id f3241b76dda919957e6293d76ac1bbe4e67d220e69768102af1c42d312df0a4f
Sep 04 22:56:11.695870 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:11.695826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqggf" event={"ID":"65cf4b75-4292-41a2-a3f0-4b107a3876be","Type":"ContainerStarted","Data":"f3241b76dda919957e6293d76ac1bbe4e67d220e69768102af1c42d312df0a4f"}
Sep 04 22:56:11.697146 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:11.697121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nqzzn" event={"ID":"007ed975-6abd-4255-9a63-f450245675f1","Type":"ContainerStarted","Data":"df6adae47823f09b88ab6926a9d63757501e35c8bca19d7d40f909177b8f5eea"}
Sep 04 22:56:12.709302 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:12.709260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6" event={"ID":"7778dbc0-aa1d-4071-bcb1-baec439b937a","Type":"ContainerStarted","Data":"9116d27ac7d7d5232e6b9d214ff8e42a50da57c142a55cd950dc0555dc3fba1c"}
Sep 04 22:56:12.710146 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:12.710123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:14.718282 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.718240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqggf" event={"ID":"65cf4b75-4292-41a2-a3f0-4b107a3876be","Type":"ContainerStarted","Data":"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"}
Sep 04 22:56:14.719669 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.719645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nqzzn" event={"ID":"007ed975-6abd-4255-9a63-f450245675f1","Type":"ContainerStarted","Data":"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"}
Sep 04 22:56:14.732927 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.732880    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-mqggf" podStartSLOduration=1.7517138920000002 podStartE2EDuration="4.732863094s" totalImagesPullingTime="2.981149202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:56:14.731610539 +0000 UTC m=+658.902685162" watchObservedRunningTime="2026-09-04 22:56:14.732863094 +0000 UTC m=+658.903937710"
Sep 04 22:56:14.733241 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.733215    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6" podStartSLOduration=3.53110179 podStartE2EDuration="5.733208577s" totalImagesPullingTime="2.202106787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:56:12.730927651 +0000 UTC m=+656.902002264" watchObservedRunningTime="2026-09-04 22:56:14.733208577 +0000 UTC m=+658.904283189"
Sep 04 22:56:14.746116 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.746052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-nqzzn" podStartSLOduration=1.636305441 podStartE2EDuration="4.746038311s" totalImagesPullingTime="3.10973287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:56:14.746014183 +0000 UTC m=+658.917088805" watchObservedRunningTime="2026-09-04 22:56:14.746038311 +0000 UTC m=+658.917112926"
Sep 04 22:56:14.770559 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:14.770528    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:16.726885 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:16.726847    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-nqzzn" podUID="007ed975-6abd-4255-9a63-f450245675f1" containerName="authorino" containerID="cri-o://dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2" gracePeriod=30
Sep 04 22:56:16.970430 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:16.970405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:17.073392 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.073308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7c54\" (UniqueName: \"kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54\") pod \"007ed975-6abd-4255-9a63-f450245675f1\" (UID: \"007ed975-6abd-4255-9a63-f450245675f1\") "
Sep 04 22:56:17.075407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.075381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54" (OuterVolumeSpecName: "kube-api-access-m7c54") pod "007ed975-6abd-4255-9a63-f450245675f1" (UID: "007ed975-6abd-4255-9a63-f450245675f1"). InnerVolumeSpecName "kube-api-access-m7c54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:56:17.174823 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.174791    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c54\" (UniqueName: \"kubernetes.io/projected/007ed975-6abd-4255-9a63-f450245675f1-kube-api-access-m7c54\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:56:17.731475 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.731436    2577 generic.go:356] "Generic (PLEG): container finished" podID="007ed975-6abd-4255-9a63-f450245675f1" containerID="dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2" exitCode=0
Sep 04 22:56:17.731903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.731485    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-nqzzn"
Sep 04 22:56:17.731903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.731518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nqzzn" event={"ID":"007ed975-6abd-4255-9a63-f450245675f1","Type":"ContainerDied","Data":"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"}
Sep 04 22:56:17.731903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.731562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-nqzzn" event={"ID":"007ed975-6abd-4255-9a63-f450245675f1","Type":"ContainerDied","Data":"df6adae47823f09b88ab6926a9d63757501e35c8bca19d7d40f909177b8f5eea"}
Sep 04 22:56:17.731903 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.731580    2577 scope.go:117] "RemoveContainer" containerID="dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"
Sep 04 22:56:17.739552 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.739533    2577 scope.go:117] "RemoveContainer" containerID="dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"
Sep 04 22:56:17.739839 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:56:17.739821    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2\": container with ID starting with dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2 not found: ID does not exist" containerID="dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"
Sep 04 22:56:17.739891 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.739847    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2"} err="failed to get container status \"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2\": rpc error: code = NotFound desc = could not find container \"dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2\": container with ID starting with dba7f2e47a78f4fa61819727735bb89f70e91f7267f7767b08bf13cbc34182f2 not found: ID does not exist"
Sep 04 22:56:17.761372 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.761348    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:17.766274 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:17.766253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-nqzzn"]
Sep 04 22:56:18.325731 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:18.325699    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007ed975-6abd-4255-9a63-f450245675f1" path="/var/lib/kubelet/pods/007ed975-6abd-4255-9a63-f450245675f1/volumes"
Sep 04 22:56:24.721494 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:24.721461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"
Sep 04 22:56:39.955570 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.955534    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:39.955965 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.955812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007ed975-6abd-4255-9a63-f450245675f1" containerName="authorino"
Sep 04 22:56:39.955965 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.955824    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="007ed975-6abd-4255-9a63-f450245675f1" containerName="authorino"
Sep 04 22:56:39.955965 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.955903    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="007ed975-6abd-4255-9a63-f450245675f1" containerName="authorino"
Sep 04 22:56:39.959973 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.959958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:39.965473 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:39.965450    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:40.056091 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.056046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfb7s\" (UniqueName: \"kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s\") pod \"authorino-5dc7ccb5f-rvb5t\" (UID: \"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c\") " pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:40.157332 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.157291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfb7s\" (UniqueName: \"kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s\") pod \"authorino-5dc7ccb5f-rvb5t\" (UID: \"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c\") " pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:40.166002 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.165972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfb7s\" (UniqueName: \"kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s\") pod \"authorino-5dc7ccb5f-rvb5t\" (UID: \"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c\") " pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:40.196869 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.196833    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:40.197052 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.197040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:40.224077 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.224047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74d6c4bf5b-g9j7n"]
Sep 04 22:56:40.227268 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.227245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.236142 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.236112    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d6c4bf5b-g9j7n"]
Sep 04 22:56:40.258390 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.258360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbdz\" (UniqueName: \"kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz\") pod \"authorino-74d6c4bf5b-g9j7n\" (UID: \"0106c928-f8b5-486f-aedb-cbbed68ad867\") " pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.296651 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.296615    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d6c4bf5b-g9j7n"]
Sep 04 22:56:40.296919 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:56:40.296895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6nbdz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n" podUID="0106c928-f8b5-486f-aedb-cbbed68ad867"
Sep 04 22:56:40.332520 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.332482    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 22:56:40.335485 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.335464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.337822 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.337802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 04 22:56:40.343917 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.343894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 22:56:40.358841 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.358812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.358977 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.358863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lqjw\" (UniqueName: \"kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.358977 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.358892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbdz\" (UniqueName: \"kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz\") pod \"authorino-74d6c4bf5b-g9j7n\" (UID: \"0106c928-f8b5-486f-aedb-cbbed68ad867\") " pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.366172 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.366145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbdz\" (UniqueName: \"kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz\") pod \"authorino-74d6c4bf5b-g9j7n\" (UID: \"0106c928-f8b5-486f-aedb-cbbed68ad867\") " pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.459320 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.459236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.459320 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.459296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lqjw\" (UniqueName: \"kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.461618 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.461592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.467257 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.467228    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:40.469910 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.469889    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lqjw\" (UniqueName: \"kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw\") pod \"authorino-5cc7f6485d-s9cfg\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") " pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.470764 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:56:40.470723    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fd9cb3e_dcbf_4115_bc59_219e5d7d808c.slice/crio-550e4663eaf874447945a11b92bd07ccccd06cdb28733911447392d93b114ff8 WatchSource:0}: Error finding container 550e4663eaf874447945a11b92bd07ccccd06cdb28733911447392d93b114ff8: Status 404 returned error can't find the container with id 550e4663eaf874447945a11b92bd07ccccd06cdb28733911447392d93b114ff8
Sep 04 22:56:40.644941 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.644907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:56:40.815936 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.815846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" event={"ID":"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c","Type":"ContainerStarted","Data":"550e4663eaf874447945a11b92bd07ccccd06cdb28733911447392d93b114ff8"}
Sep 04 22:56:40.815936 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.815860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.822266 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.822245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:40.862437 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.862406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbdz\" (UniqueName: \"kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz\") pod \"0106c928-f8b5-486f-aedb-cbbed68ad867\" (UID: \"0106c928-f8b5-486f-aedb-cbbed68ad867\") "
Sep 04 22:56:40.864505 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.864482    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz" (OuterVolumeSpecName: "kube-api-access-6nbdz") pod "0106c928-f8b5-486f-aedb-cbbed68ad867" (UID: "0106c928-f8b5-486f-aedb-cbbed68ad867"). InnerVolumeSpecName "kube-api-access-6nbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:56:40.911905 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.911872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 22:56:40.914914 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:56:40.914887    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f384eb3_8b02_4d34_9d4a_8f8df576f9e0.slice/crio-c2e143270fabe361438a3757ea35f11f9e34b7d4ddac704fa1ebe3ba1210fa07 WatchSource:0}: Error finding container c2e143270fabe361438a3757ea35f11f9e34b7d4ddac704fa1ebe3ba1210fa07: Status 404 returned error can't find the container with id c2e143270fabe361438a3757ea35f11f9e34b7d4ddac704fa1ebe3ba1210fa07
Sep 04 22:56:40.963653 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:40.963630    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbdz\" (UniqueName: \"kubernetes.io/projected/0106c928-f8b5-486f-aedb-cbbed68ad867-kube-api-access-6nbdz\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:56:41.821061 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.821021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" event={"ID":"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c","Type":"ContainerStarted","Data":"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"}
Sep 04 22:56:41.821239 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.821105    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" podUID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" containerName="authorino" containerID="cri-o://4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b" gracePeriod=30
Sep 04 22:56:41.822636 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.822618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d6c4bf5b-g9j7n"
Sep 04 22:56:41.822756 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.822613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" event={"ID":"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0","Type":"ContainerStarted","Data":"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"}
Sep 04 22:56:41.822756 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.822723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" event={"ID":"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0","Type":"ContainerStarted","Data":"c2e143270fabe361438a3757ea35f11f9e34b7d4ddac704fa1ebe3ba1210fa07"}
Sep 04 22:56:41.835627 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.835587    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" podStartSLOduration=2.383146294 podStartE2EDuration="2.835574958s" totalImagesPullingTime="452.428664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:56:41.834266364 +0000 UTC m=+686.005340976" watchObservedRunningTime="2026-09-04 22:56:41.835574958 +0000 UTC m=+686.006649570"
Sep 04 22:56:41.848812 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.848772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" podStartSLOduration=1.524175573 podStartE2EDuration="1.848757957s" totalImagesPullingTime="324.582384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:56:41.847956379 +0000 UTC m=+686.019030991" watchObservedRunningTime="2026-09-04 22:56:41.848757957 +0000 UTC m=+686.019832566"
Sep 04 22:56:41.872214 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.872183    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:41.872379 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.872356    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-mqggf" podUID="65cf4b75-4292-41a2-a3f0-4b107a3876be" containerName="authorino" containerID="cri-o://882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a" gracePeriod=30
Sep 04 22:56:41.936278 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.936247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d6c4bf5b-g9j7n"]
Sep 04 22:56:41.939266 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:41.939239    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74d6c4bf5b-g9j7n"]
Sep 04 22:56:42.128045 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.128018    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:42.135896 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.135872    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:42.175301 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.175222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfb7s\" (UniqueName: \"kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s\") pod \"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c\" (UID: \"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c\") "
Sep 04 22:56:42.175502 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.175321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xl8h\" (UniqueName: \"kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h\") pod \"65cf4b75-4292-41a2-a3f0-4b107a3876be\" (UID: \"65cf4b75-4292-41a2-a3f0-4b107a3876be\") "
Sep 04 22:56:42.177659 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.177623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h" (OuterVolumeSpecName: "kube-api-access-4xl8h") pod "65cf4b75-4292-41a2-a3f0-4b107a3876be" (UID: "65cf4b75-4292-41a2-a3f0-4b107a3876be"). InnerVolumeSpecName "kube-api-access-4xl8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:56:42.177819 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.177797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s" (OuterVolumeSpecName: "kube-api-access-vfb7s") pod "4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" (UID: "4fd9cb3e-dcbf-4115-bc59-219e5d7d808c"). InnerVolumeSpecName "kube-api-access-vfb7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:56:42.276374 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.276335    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xl8h\" (UniqueName: \"kubernetes.io/projected/65cf4b75-4292-41a2-a3f0-4b107a3876be-kube-api-access-4xl8h\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:56:42.276374 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.276366    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfb7s\" (UniqueName: \"kubernetes.io/projected/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c-kube-api-access-vfb7s\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 22:56:42.324200 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.324173    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0106c928-f8b5-486f-aedb-cbbed68ad867" path="/var/lib/kubelet/pods/0106c928-f8b5-486f-aedb-cbbed68ad867/volumes"
Sep 04 22:56:42.827397 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.827308    2577 generic.go:356] "Generic (PLEG): container finished" podID="65cf4b75-4292-41a2-a3f0-4b107a3876be" containerID="882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a" exitCode=0
Sep 04 22:56:42.827397 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.827348    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-mqggf"
Sep 04 22:56:42.827581 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.827393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqggf" event={"ID":"65cf4b75-4292-41a2-a3f0-4b107a3876be","Type":"ContainerDied","Data":"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"}
Sep 04 22:56:42.827581 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.827437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-mqggf" event={"ID":"65cf4b75-4292-41a2-a3f0-4b107a3876be","Type":"ContainerDied","Data":"f3241b76dda919957e6293d76ac1bbe4e67d220e69768102af1c42d312df0a4f"}
Sep 04 22:56:42.827581 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.827455    2577 scope.go:117] "RemoveContainer" containerID="882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"
Sep 04 22:56:42.828798 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.828666    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" containerID="4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b" exitCode=0
Sep 04 22:56:42.828798 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.828700    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t"
Sep 04 22:56:42.828798 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.828758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" event={"ID":"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c","Type":"ContainerDied","Data":"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"}
Sep 04 22:56:42.828798 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.828793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-rvb5t" event={"ID":"4fd9cb3e-dcbf-4115-bc59-219e5d7d808c","Type":"ContainerDied","Data":"550e4663eaf874447945a11b92bd07ccccd06cdb28733911447392d93b114ff8"}
Sep 04 22:56:42.835422 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.835405    2577 scope.go:117] "RemoveContainer" containerID="882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"
Sep 04 22:56:42.835666 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:56:42.835645    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a\": container with ID starting with 882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a not found: ID does not exist" containerID="882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"
Sep 04 22:56:42.835869 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.835672    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a"} err="failed to get container status \"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a\": rpc error: code = NotFound desc = could not find container \"882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a\": container with ID starting with 882214ac1ba728fa62ea7a6a18adf7d8c0e2286adbabdeda6e100de29ddca92a not found: ID does not exist"
Sep 04 22:56:42.835869 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.835688    2577 scope.go:117] "RemoveContainer" containerID="4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"
Sep 04 22:56:42.842404 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.842388    2577 scope.go:117] "RemoveContainer" containerID="4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"
Sep 04 22:56:42.842679 ip-10-0-134-172 kubenswrapper[2577]: E0904 22:56:42.842659    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b\": container with ID starting with 4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b not found: ID does not exist" containerID="4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"
Sep 04 22:56:42.842828 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.842685    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b"} err="failed to get container status \"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b\": rpc error: code = NotFound desc = could not find container \"4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b\": container with ID starting with 4770e20344bd9b56403a69bfbd1d862e07e86b898bb3c805a0bf94f4b4e5135b not found: ID does not exist"
Sep 04 22:56:42.844445 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.844427    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:42.848158 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.848139    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-mqggf"]
Sep 04 22:56:42.857777 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.857747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:42.861287 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:42.861266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-rvb5t"]
Sep 04 22:56:44.324747 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:44.324701    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" path="/var/lib/kubelet/pods/4fd9cb3e-dcbf-4115-bc59-219e5d7d808c/volumes"
Sep 04 22:56:44.325109 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:56:44.325042    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65cf4b75-4292-41a2-a3f0-4b107a3876be" path="/var/lib/kubelet/pods/65cf4b75-4292-41a2-a3f0-4b107a3876be/volumes"
Sep 04 22:57:22.272369 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:57:22.272329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:57:40.094613 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:57:40.094573    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:57:48.190260 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:57:48.190223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:57:51.787590 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:57:51.787551    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:57:58.901000 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:57:58.900956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:58:04.879661 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.879627    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"]
Sep 04 22:58:04.880067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880047    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65cf4b75-4292-41a2-a3f0-4b107a3876be" containerName="authorino"
Sep 04 22:58:04.880067 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880065    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="65cf4b75-4292-41a2-a3f0-4b107a3876be" containerName="authorino"
Sep 04 22:58:04.880178 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880097    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" containerName="authorino"
Sep 04 22:58:04.880178 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880108    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" containerName="authorino"
Sep 04 22:58:04.880264 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880179    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="65cf4b75-4292-41a2-a3f0-4b107a3876be" containerName="authorino"
Sep 04 22:58:04.880264 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.880192    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd9cb3e-dcbf-4115-bc59-219e5d7d808c" containerName="authorino"
Sep 04 22:58:04.883559 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.883537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.886115 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.886084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 04 22:58:04.886908 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.886885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vktld\""
Sep 04 22:58:04.887016 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.886897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 04 22:58:04.887016 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.886930    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 04 22:58:04.893128 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.893101    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"]
Sep 04 22:58:04.919165 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398e4815-caf6-49b2-970b-4eb23ff9d86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.919342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.919342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.919342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.919342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:04.919342 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:04.919330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s89bc\" (UniqueName: \"kubernetes.io/projected/398e4815-caf6-49b2-970b-4eb23ff9d86c-kube-api-access-s89bc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.019904 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.019869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.019904 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.019907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020086 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.019926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020086 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.019947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s89bc\" (UniqueName: \"kubernetes.io/projected/398e4815-caf6-49b2-970b-4eb23ff9d86c-kube-api-access-s89bc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020086 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.020002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398e4815-caf6-49b2-970b-4eb23ff9d86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020086 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.020041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020377 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.020355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020453 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.020400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.020453 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.020429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.022097 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.022078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398e4815-caf6-49b2-970b-4eb23ff9d86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.022458 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.022437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398e4815-caf6-49b2-970b-4eb23ff9d86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.029135 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.029104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s89bc\" (UniqueName: \"kubernetes.io/projected/398e4815-caf6-49b2-970b-4eb23ff9d86c-kube-api-access-s89bc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp\" (UID: \"398e4815-caf6-49b2-970b-4eb23ff9d86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.195167 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.195070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:05.463456 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:05.463433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"]
Sep 04 22:58:05.466058 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:58:05.466027    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398e4815_caf6_49b2_970b_4eb23ff9d86c.slice/crio-8108666a23f0264aab8a22f7f689b75c603cc8b93f596e0f71db45bbbcca9e2c WatchSource:0}: Error finding container 8108666a23f0264aab8a22f7f689b75c603cc8b93f596e0f71db45bbbcca9e2c: Status 404 returned error can't find the container with id 8108666a23f0264aab8a22f7f689b75c603cc8b93f596e0f71db45bbbcca9e2c
Sep 04 22:58:06.156857 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:06.156813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp" event={"ID":"398e4815-caf6-49b2-970b-4eb23ff9d86c","Type":"ContainerStarted","Data":"8108666a23f0264aab8a22f7f689b75c603cc8b93f596e0f71db45bbbcca9e2c"}
Sep 04 22:58:06.287062 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:06.287026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:58:07.162371 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:07.162333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp" event={"ID":"398e4815-caf6-49b2-970b-4eb23ff9d86c","Type":"ContainerStarted","Data":"b3e22e810865277d92da2cc618483be9ed568ef095107c85c617518b14579a91"}
Sep 04 22:58:07.180209 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:07.180156    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp" podStartSLOduration=2.228962323 podStartE2EDuration="3.180142145s" totalImagesPullingTime="951.179822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:58:07.178831473 +0000 UTC m=+771.349906085" watchObservedRunningTime="2026-09-04 22:58:07.180142145 +0000 UTC m=+771.351216757"
Sep 04 22:58:14.694687 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:14.694648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:58:15.195407 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:15.195379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:15.195587 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:15.195433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:15.208460 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:15.208432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:16.209482 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:16.209444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp"
Sep 04 22:58:22.585501 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.585459    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"]
Sep 04 22:58:22.588681 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.588661    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.591346 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.591324    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 04 22:58:22.598266 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.598245    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"]
Sep 04 22:58:22.667070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.667070 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.667280 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vdz\" (UniqueName: \"kubernetes.io/projected/9dab545d-8d66-4541-a517-1639170f09fc-kube-api-access-h2vdz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.667280 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab545d-8d66-4541-a517-1639170f09fc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.667280 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.667280 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.667239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768204 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768288    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vdz\" (UniqueName: \"kubernetes.io/projected/9dab545d-8d66-4541-a517-1639170f09fc-kube-api-access-h2vdz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768356 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab545d-8d66-4541-a517-1639170f09fc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768733 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768733 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.768868 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.768763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.770447 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.770421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dab545d-8d66-4541-a517-1639170f09fc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.770882 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.770859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dab545d-8d66-4541-a517-1639170f09fc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.776226 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.776204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vdz\" (UniqueName: \"kubernetes.io/projected/9dab545d-8d66-4541-a517-1639170f09fc-kube-api-access-h2vdz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8\" (UID: \"9dab545d-8d66-4541-a517-1639170f09fc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:22.897479 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:22.897386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:23.163051 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:23.163022    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"]
Sep 04 22:58:23.165563 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:58:23.165537    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dab545d_8d66_4541_a517_1639170f09fc.slice/crio-00694ec1ce9460c96d16883155bffb475d25cf4fb0935c8aaa1c28a048f68e0e WatchSource:0}: Error finding container 00694ec1ce9460c96d16883155bffb475d25cf4fb0935c8aaa1c28a048f68e0e: Status 404 returned error can't find the container with id 00694ec1ce9460c96d16883155bffb475d25cf4fb0935c8aaa1c28a048f68e0e
Sep 04 22:58:23.167157 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:23.167134    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:58:23.227162 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:23.227120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8" event={"ID":"9dab545d-8d66-4541-a517-1639170f09fc","Type":"ContainerStarted","Data":"00694ec1ce9460c96d16883155bffb475d25cf4fb0935c8aaa1c28a048f68e0e"}
Sep 04 22:58:24.237753 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:24.237707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8" event={"ID":"9dab545d-8d66-4541-a517-1639170f09fc","Type":"ContainerStarted","Data":"cd97a6f66aa2373e8b6b524a0bb967b1b9212b720f795c337846a56308f73c15"}
Sep 04 22:58:24.255732 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:24.255684    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8" podStartSLOduration=2.069100064 podStartE2EDuration="2.255671254s" totalImagesPullingTime="186.57119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:58:24.254395017 +0000 UTC m=+788.425469629" watchObservedRunningTime="2026-09-04 22:58:24.255671254 +0000 UTC m=+788.426745866"
Sep 04 22:58:24.489360 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:24.489270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 22:58:32.898526 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:32.898488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:32.899025 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:32.898549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:32.910876 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:32.910854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:58:33.282914 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:58:33.282834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8"
Sep 04 22:59:57.752266 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.752228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6784dc565f-mq2pt"]
Sep 04 22:59:57.755621 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.755594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.764142 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.764113    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6784dc565f-mq2pt"]
Sep 04 22:59:57.846634 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.846598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnpv\" (UniqueName: \"kubernetes.io/projected/e099cb13-85be-4894-8f7f-2d729efb4fcf-kube-api-access-qdnpv\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.846819 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.846648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e099cb13-85be-4894-8f7f-2d729efb4fcf-tls-cert\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.947277 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.947233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e099cb13-85be-4894-8f7f-2d729efb4fcf-tls-cert\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.947436 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.947336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnpv\" (UniqueName: \"kubernetes.io/projected/e099cb13-85be-4894-8f7f-2d729efb4fcf-kube-api-access-qdnpv\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.949873 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.949850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e099cb13-85be-4894-8f7f-2d729efb4fcf-tls-cert\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:57.954836 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:57.954806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnpv\" (UniqueName: \"kubernetes.io/projected/e099cb13-85be-4894-8f7f-2d729efb4fcf-kube-api-access-qdnpv\") pod \"authorino-6784dc565f-mq2pt\" (UID: \"e099cb13-85be-4894-8f7f-2d729efb4fcf\") " pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:58.065023 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:58.064942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6784dc565f-mq2pt"
Sep 04 22:59:58.332683 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:58.332518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6784dc565f-mq2pt"]
Sep 04 22:59:58.335448 ip-10-0-134-172 kubenswrapper[2577]: W0904 22:59:58.335418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode099cb13_85be_4894_8f7f_2d729efb4fcf.slice/crio-31c3e82cbde6623b18ad8402f2a8f730feb5e50c071d923c63137d62835d0ced WatchSource:0}: Error finding container 31c3e82cbde6623b18ad8402f2a8f730feb5e50c071d923c63137d62835d0ced: Status 404 returned error can't find the container with id 31c3e82cbde6623b18ad8402f2a8f730feb5e50c071d923c63137d62835d0ced
Sep 04 22:59:58.611718 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:58.611605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6784dc565f-mq2pt" event={"ID":"e099cb13-85be-4894-8f7f-2d729efb4fcf","Type":"ContainerStarted","Data":"31c3e82cbde6623b18ad8402f2a8f730feb5e50c071d923c63137d62835d0ced"}
Sep 04 22:59:59.619162 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.619111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6784dc565f-mq2pt" event={"ID":"e099cb13-85be-4894-8f7f-2d729efb4fcf","Type":"ContainerStarted","Data":"ef57eeaaf8e443cac7d1b65844c815b8759fa0e1e5a57f2ebc1682ee4ad9eb08"}
Sep 04 22:59:59.636952 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.636893    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6784dc565f-mq2pt" podStartSLOduration=1.728291566 podStartE2EDuration="2.636877061s" totalImagesPullingTime="908.585495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:59:59.635056883 +0000 UTC m=+883.806131495" watchObservedRunningTime="2026-09-04 22:59:59.636877061 +0000 UTC m=+883.807951673"
Sep 04 22:59:59.664170 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.664133    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 22:59:59.664359 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.664318    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" podUID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" containerName="authorino" containerID="cri-o://df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d" gracePeriod=30
Sep 04 22:59:59.926934 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.926909    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 22:59:59.963508 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.963474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lqjw\" (UniqueName: \"kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw\") pod \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") "
Sep 04 22:59:59.963675 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.963605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert\") pod \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\" (UID: \"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0\") "
Sep 04 22:59:59.965641 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.965596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw" (OuterVolumeSpecName: "kube-api-access-6lqjw") pod "9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" (UID: "9f384eb3-8b02-4d34-9d4a-8f8df576f9e0"). InnerVolumeSpecName "kube-api-access-6lqjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:59:59.973936 ip-10-0-134-172 kubenswrapper[2577]: I0904 22:59:59.973900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" (UID: "9f384eb3-8b02-4d34-9d4a-8f8df576f9e0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:00:00.064628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.064593    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-tls-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:00:00.064628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.064625    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lqjw\" (UniqueName: \"kubernetes.io/projected/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0-kube-api-access-6lqjw\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:00:00.624558 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.624523    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" containerID="df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d" exitCode=0
Sep 04 23:00:00.624973 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.624566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg"
Sep 04 23:00:00.624973 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.624618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" event={"ID":"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0","Type":"ContainerDied","Data":"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"}
Sep 04 23:00:00.624973 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.624668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5cc7f6485d-s9cfg" event={"ID":"9f384eb3-8b02-4d34-9d4a-8f8df576f9e0","Type":"ContainerDied","Data":"c2e143270fabe361438a3757ea35f11f9e34b7d4ddac704fa1ebe3ba1210fa07"}
Sep 04 23:00:00.624973 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.624696    2577 scope.go:117] "RemoveContainer" containerID="df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"
Sep 04 23:00:00.638845 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.638732    2577 scope.go:117] "RemoveContainer" containerID="df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"
Sep 04 23:00:00.639209 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:00:00.639188    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d\": container with ID starting with df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d not found: ID does not exist" containerID="df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"
Sep 04 23:00:00.639285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.639217    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d"} err="failed to get container status \"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d\": rpc error: code = NotFound desc = could not find container \"df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d\": container with ID starting with df60a3749ae1dfbca7ab24fd3966babc636272d57fbb04b1a183c7ee48f3823d not found: ID does not exist"
Sep 04 23:00:00.659317 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.659274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 23:00:00.660355 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:00.660334    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5cc7f6485d-s9cfg"]
Sep 04 23:00:02.325391 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:02.325354    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" path="/var/lib/kubelet/pods/9f384eb3-8b02-4d34-9d4a-8f8df576f9e0/volumes"
Sep 04 23:00:19.429706 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.429666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-140135280585008"]
Sep 04 23:00:19.430094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.429980    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" containerName="authorino"
Sep 04 23:00:19.430094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.429993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" containerName="authorino"
Sep 04 23:00:19.430094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.430044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f384eb3-8b02-4d34-9d4a-8f8df576f9e0" containerName="authorino"
Sep 04 23:00:19.433494 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.433474    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:19.436361 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.436338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s76bh\""
Sep 04 23:00:19.437870 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.437845    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-140135280585008"]
Sep 04 23:00:19.539131 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.539085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g6q\" (UniqueName: \"kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q\") pod \"test-curl-8011-140135280585008\" (UID: \"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69\") " pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:19.639853 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.639814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g6q\" (UniqueName: \"kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q\") pod \"test-curl-8011-140135280585008\" (UID: \"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69\") " pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:19.647171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.647145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g6q\" (UniqueName: \"kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q\") pod \"test-curl-8011-140135280585008\" (UID: \"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69\") " pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:19.744094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:19.744005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:20.012926 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:20.012897    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-140135280585008"]
Sep 04 23:00:20.015578 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:20.015540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c2e5b11_e6f3_4ddd_978a_b72c387c5f69.slice/crio-c6deeed03b3b4682abe3b17f984805e6fabc7fedeb714c85f3c68ea9ddb25ad7 WatchSource:0}: Error finding container c6deeed03b3b4682abe3b17f984805e6fabc7fedeb714c85f3c68ea9ddb25ad7: Status 404 returned error can't find the container with id c6deeed03b3b4682abe3b17f984805e6fabc7fedeb714c85f3c68ea9ddb25ad7
Sep 04 23:00:20.708060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:20.708025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-140135280585008" event={"ID":"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69","Type":"ContainerStarted","Data":"c6deeed03b3b4682abe3b17f984805e6fabc7fedeb714c85f3c68ea9ddb25ad7"}
Sep 04 23:00:21.156065 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:21.156015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:00:22.718162 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:22.718116    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" containerID="3e8c3ab591d8c8d4fb7c8863a65261554a000f60a988139dfede9007b52246af" exitCode=0
Sep 04 23:00:22.718515 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:22.718174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-140135280585008" event={"ID":"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69","Type":"ContainerDied","Data":"3e8c3ab591d8c8d4fb7c8863a65261554a000f60a988139dfede9007b52246af"}
Sep 04 23:00:23.849638 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.849608    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:23.882210 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.882180    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8011-140135280585008_7c2e5b11-e6f3-4ddd-978a-b72c387c5f69/test-curl-8011-140135280585008/0.log"
Sep 04 23:00:23.903458 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.903419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-140135280585008"]
Sep 04 23:00:23.905320 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.905290    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-140135280585008"]
Sep 04 23:00:23.981382 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.981308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8g6q\" (UniqueName: \"kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q\") pod \"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69\" (UID: \"7c2e5b11-e6f3-4ddd-978a-b72c387c5f69\") "
Sep 04 23:00:23.983509 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:23.983483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q" (OuterVolumeSpecName: "kube-api-access-h8g6q") pod "7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" (UID: "7c2e5b11-e6f3-4ddd-978a-b72c387c5f69"). InnerVolumeSpecName "kube-api-access-h8g6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:00:24.023126 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.023091    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-140135280583984"]
Sep 04 23:00:24.023376 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.023364    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" containerName="test-curl-8011-140135280585008"
Sep 04 23:00:24.023418 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.023378    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" containerName="test-curl-8011-140135280585008"
Sep 04 23:00:24.023449 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.023436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" containerName="test-curl-8011-140135280585008"
Sep 04 23:00:24.026260 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.026244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:24.029782 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.029755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-140135280583984"]
Sep 04 23:00:24.082872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.082832    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8g6q\" (UniqueName: \"kubernetes.io/projected/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69-kube-api-access-h8g6q\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:00:24.183989 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.183949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh5r\" (UniqueName: \"kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r\") pod \"test-curl-8011-140135280583984\" (UID: \"48c97a75-92d3-4225-8b48-19937b4272b7\") " pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:24.284719 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.284624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh5r\" (UniqueName: \"kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r\") pod \"test-curl-8011-140135280583984\" (UID: \"48c97a75-92d3-4225-8b48-19937b4272b7\") " pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:24.292688 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.292659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh5r\" (UniqueName: \"kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r\") pod \"test-curl-8011-140135280583984\" (UID: \"48c97a75-92d3-4225-8b48-19937b4272b7\") " pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:24.324965 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.324931    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2e5b11-e6f3-4ddd-978a-b72c387c5f69" path="/var/lib/kubelet/pods/7c2e5b11-e6f3-4ddd-978a-b72c387c5f69/volumes"
Sep 04 23:00:24.335353 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.335332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:24.601222 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.601186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-140135280583984"]
Sep 04 23:00:24.603853 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:24.603823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48c97a75_92d3_4225_8b48_19937b4272b7.slice/crio-4d69df433566f562cf42d9c09a75694390d6269eed48140c927e7c7649652d70 WatchSource:0}: Error finding container 4d69df433566f562cf42d9c09a75694390d6269eed48140c927e7c7649652d70: Status 404 returned error can't find the container with id 4d69df433566f562cf42d9c09a75694390d6269eed48140c927e7c7649652d70
Sep 04 23:00:24.727214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.727187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280585008"
Sep 04 23:00:24.727399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.727240    2577 scope.go:117] "RemoveContainer" containerID="3e8c3ab591d8c8d4fb7c8863a65261554a000f60a988139dfede9007b52246af"
Sep 04 23:00:24.728515 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:24.728397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-140135280583984" event={"ID":"48c97a75-92d3-4225-8b48-19937b4272b7","Type":"ContainerStarted","Data":"4d69df433566f562cf42d9c09a75694390d6269eed48140c927e7c7649652d70"}
Sep 04 23:00:25.734152 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:25.734118    2577 generic.go:356] "Generic (PLEG): container finished" podID="48c97a75-92d3-4225-8b48-19937b4272b7" containerID="a92874dc4e850f001701880ddae7762b71aa0a4298920723fddc2de3b363f8a2" exitCode=0
Sep 04 23:00:25.734540 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:25.734163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-140135280583984" event={"ID":"48c97a75-92d3-4225-8b48-19937b4272b7","Type":"ContainerDied","Data":"a92874dc4e850f001701880ddae7762b71aa0a4298920723fddc2de3b363f8a2"}
Sep 04 23:00:26.886297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:26.886261    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:26.905152 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:26.905125    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8011-140135280583984_48c97a75-92d3-4225-8b48-19937b4272b7/test-curl-8011-140135280583984/0.log"
Sep 04 23:00:26.928061 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:26.928024    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-140135280583984"]
Sep 04 23:00:26.932334 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:26.932302    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-140135280583984"]
Sep 04 23:00:27.012171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.012072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kh5r\" (UniqueName: \"kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r\") pod \"48c97a75-92d3-4225-8b48-19937b4272b7\" (UID: \"48c97a75-92d3-4225-8b48-19937b4272b7\") "
Sep 04 23:00:27.014166 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.014138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r" (OuterVolumeSpecName: "kube-api-access-6kh5r") pod "48c97a75-92d3-4225-8b48-19937b4272b7" (UID: "48c97a75-92d3-4225-8b48-19937b4272b7"). InnerVolumeSpecName "kube-api-access-6kh5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:00:27.112810 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.112730    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh5r\" (UniqueName: \"kubernetes.io/projected/48c97a75-92d3-4225-8b48-19937b4272b7-kube-api-access-6kh5r\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:00:27.746769 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.746725    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d69df433566f562cf42d9c09a75694390d6269eed48140c927e7c7649652d70"
Sep 04 23:00:27.746942 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.746784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:27.758693 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:00:27.758649    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-140135280583984\" is forbidden: User \"system:node:ip-10-0-134-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-172.ec2.internal' and this object" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:27.759886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.759862    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:00:27.760180 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.760165    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" containerName="test-curl-8011-140135280583984"
Sep 04 23:00:27.760251 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.760183    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" containerName="test-curl-8011-140135280583984"
Sep 04 23:00:27.760307 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.760269    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" containerName="test-curl-8011-140135280583984"
Sep 04 23:00:27.763167 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.763146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.765829 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:00:27.765787    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-140135280583984\" is forbidden: User \"system:node:ip-10-0-134-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-172.ec2.internal' and this object" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" pod="opendatahub/test-curl-8011-140135280583984"
Sep 04 23:00:27.765938 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.765891    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 23:00:27.765938 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.765913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-63d928-openshift-default-dockercfg-262pd\""
Sep 04 23:00:27.776863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.776834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:00:27.920948 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.920891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.920948 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.920956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gff6\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:27.921431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:27.921336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022676 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022676 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022676 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gff6\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.022980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.022994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.023093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.023122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023287 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.023219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023402 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.023386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.023547 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.023531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.025085 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.025060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.025209 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.025173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.032702 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.032677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.032865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.032843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gff6\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6\") pod \"e2e-autores-63d928-openshift-default-6565cd9666-w7r79\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") " pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.073498 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.073457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:28.324671 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.324584    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c97a75-92d3-4225-8b48-19937b4272b7" path="/var/lib/kubelet/pods/48c97a75-92d3-4225-8b48-19937b4272b7/volumes"
Sep 04 23:00:28.355022 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.354995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:00:28.357224 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:28.357196    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d629a3_edc0_410d_aa4a_44cb056bdfb5.slice/crio-81a20c23bc6fdf32b6589a0cabcc126e7fe1bb4a0e6f8d73bdd45df1ae0cbcc5 WatchSource:0}: Error finding container 81a20c23bc6fdf32b6589a0cabcc126e7fe1bb4a0e6f8d73bdd45df1ae0cbcc5: Status 404 returned error can't find the container with id 81a20c23bc6fdf32b6589a0cabcc126e7fe1bb4a0e6f8d73bdd45df1ae0cbcc5
Sep 04 23:00:28.752379 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:28.752347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerStarted","Data":"81a20c23bc6fdf32b6589a0cabcc126e7fe1bb4a0e6f8d73bdd45df1ae0cbcc5"}
Sep 04 23:00:30.766355 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:30.766319    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:30.766585 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:30.766409    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:30.766585 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:30.766440    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:30.995300 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:30.995265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:00:31.766799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:31.766762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerStarted","Data":"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"}
Sep 04 23:00:31.786087 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:31.786027    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podStartSLOduration=2.379207122 podStartE2EDuration="4.786008844s" totalImagesPullingTime="2.406801722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:00:31.78442869 +0000 UTC m=+915.955503331" watchObservedRunningTime="2026-09-04 23:00:31.786008844 +0000 UTC m=+915.957083455"
Sep 04 23:00:32.074225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:32.074135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:32.079099 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:32.079073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:32.771129 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:32.771091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:32.772169 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:32.772151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:36.774763 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.774713    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:00:36.778135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.778115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.780775 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.780752    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-d5be7135-kserve-self-signed-certs\""
Sep 04 23:00:36.790149 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.790123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:00:36.798857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.798816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.798980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.798863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.798980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.798890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.799080 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.798995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skj4j\" (UniqueName: \"kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.799080 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.799026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.799080 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.799059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900505 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skj4j\" (UniqueName: \"kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900505 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.900793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.900616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.901125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.901090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.901125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.901114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.901265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.901142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.902843 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.902821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.903145 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.903124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:36.908088 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:36.908065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skj4j\" (UniqueName: \"kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j\") pod \"e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") " pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:37.087303 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:37.087269    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:37.358435 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:37.358406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:00:37.361108 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:37.361080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod954dc02b_d0d0_401e_8a46_84d35cf1cf29.slice/crio-b05584afab8484aefcac3cffc133b143012777b69aa958ee7a350189cf344e85 WatchSource:0}: Error finding container b05584afab8484aefcac3cffc133b143012777b69aa958ee7a350189cf344e85: Status 404 returned error can't find the container with id b05584afab8484aefcac3cffc133b143012777b69aa958ee7a350189cf344e85
Sep 04 23:00:37.793727 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:37.793634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" event={"ID":"954dc02b-d0d0-401e-8a46-84d35cf1cf29","Type":"ContainerStarted","Data":"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"}
Sep 04 23:00:37.793727 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:37.793682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" event={"ID":"954dc02b-d0d0-401e-8a46-84d35cf1cf29","Type":"ContainerStarted","Data":"b05584afab8484aefcac3cffc133b143012777b69aa958ee7a350189cf344e85"}
Sep 04 23:00:37.808953 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:37.808896    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" podStartSLOduration=1.808878695 podStartE2EDuration="1.808878695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:00:37.808409945 +0000 UTC m=+921.979484558" watchObservedRunningTime="2026-09-04 23:00:37.808878695 +0000 UTC m=+921.979953325"
Sep 04 23:00:46.103631 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.103594    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:00:46.107082 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.107059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.109633 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.109612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4fd6f6-openshift-default-dockercfg-7cthb\""
Sep 04 23:00:46.118793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.118766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:00:46.192150 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192150 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bt6r\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.192581 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.192372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.293921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.293866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bt6r\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.294274 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.294112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.294274 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.294169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.294547 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.294415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.294547 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.294433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.294661 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.294597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.296149 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.296130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.296751 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.296719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.304717 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.304692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bt6r\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.305118 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.305092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token\") pod \"e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") " pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.421359 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.421268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:46.699834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.699796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:00:46.702327 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:46.702287    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb84129e_2316_47e5_8cbe_ab444e5f2f48.slice/crio-4a03f4363c1710a7636359c1ee1653783b74c688278a7a38332af98c286a95cc WatchSource:0}: Error finding container 4a03f4363c1710a7636359c1ee1653783b74c688278a7a38332af98c286a95cc: Status 404 returned error can't find the container with id 4a03f4363c1710a7636359c1ee1653783b74c688278a7a38332af98c286a95cc
Sep 04 23:00:46.704689 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.704655    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:46.704776 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.704730    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:46.704815 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.704791    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:46.833178 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.833139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerStarted","Data":"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"}
Sep 04 23:00:46.833345 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.833213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerStarted","Data":"4a03f4363c1710a7636359c1ee1653783b74c688278a7a38332af98c286a95cc"}
Sep 04 23:00:46.855647 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.855580    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podStartSLOduration=0.855558317 podStartE2EDuration="855.558317ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:00:46.85420004 +0000 UTC m=+931.025274654" watchObservedRunningTime="2026-09-04 23:00:46.855558317 +0000 UTC m=+931.026632930"
Sep 04 23:00:46.995797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:46.995691    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:00:47.087729 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.087684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:47.087939 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.087768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:47.102017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.101976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:47.422330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.422290    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:47.427035 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.427000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:47.839597 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.839565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:47.840510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.840483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:00:47.856808 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:47.856773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:00:48.025379 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.024037    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:46808->10.134.0.30:15021: read: connection reset by peer" start-of-body=
Sep 04 23:00:48.025379 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.024106    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": read tcp 10.134.0.2:46808->10.134.0.30:15021: read: connection reset by peer"
Sep 04 23:00:48.844563 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.844530    2577 generic.go:356] "Generic (PLEG): container finished" podID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerID="e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d" exitCode=0
Sep 04 23:00:48.844952 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.844608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerDied","Data":"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"}
Sep 04 23:00:48.845184 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.845168    2577 scope.go:117] "RemoveContainer" containerID="e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"
Sep 04 23:00:48.846163 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.846137    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:48.846215 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.846201    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:48.846248 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:48.846233    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:00:49.851231 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:49.851195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerStarted","Data":"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"}
Sep 04 23:00:50.074083 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.074038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:50.074402 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.074376    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:50.074500 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.074436    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:50.615930 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.615891    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:00:50.622293 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.622271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.624792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.624768    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4fd6f6\""
Sep 04 23:00:50.624886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.624768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4fd6f6-dockercfg-5zxgp\""
Sep 04 23:00:50.631332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.631306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:00:50.733333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.733165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.733333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.733250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd6g\" (UniqueName: \"kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.833852 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.833808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.834191 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.834168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd6g\" (UniqueName: \"kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.835514 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.835489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.843872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.843836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd6g\" (UniqueName: \"kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g\") pod \"payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:50.933126 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:50.933015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:51.074483 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:51.074250    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:51.074483 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:51.074314    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:51.218926 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:51.218893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:00:51.221290 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:00:51.221261    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a04d3b_b317_4d43_8c58_c6d1633b4fd0.slice/crio-1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3 WatchSource:0}: Error finding container 1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3: Status 404 returned error can't find the container with id 1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3
Sep 04 23:00:51.864960 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:51.864922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" event={"ID":"31a04d3b-b317-4d43-8c58-c6d1633b4fd0","Type":"ContainerStarted","Data":"1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3"}
Sep 04 23:00:52.073824 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:52.073791    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:52.074292 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:52.073855    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:53.073884 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:53.073849    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:53.074365 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:53.073912    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:54.074513 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:54.074467    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:54.074963 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:54.074541    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:54.890641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:54.890602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" event={"ID":"31a04d3b-b317-4d43-8c58-c6d1633b4fd0","Type":"ContainerStarted","Data":"8a6b5c39e3fad9e7e96a4e3e22958185357a5eed43b6e0d703c9f593905d034f"}
Sep 04 23:00:54.890853 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:54.890689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:00:54.906770 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:54.906704    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" podStartSLOduration=1.662977935 podStartE2EDuration="4.906687614s" totalImagesPullingTime="3.243709679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:00:54.90522334 +0000 UTC m=+939.076297952" watchObservedRunningTime="2026-09-04 23:00:54.906687614 +0000 UTC m=+939.077762250"
Sep 04 23:00:55.073987 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:55.073948    2577 patch_prober.go:28] interesting pod/e2e-autores-63d928-openshift-default-6565cd9666-w7r79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 04 23:00:55.074166 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:55.074031    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 04 23:00:56.078054 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:56.078024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:56.078539 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:56.078293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:00:56.079072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:00:56.079054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:01:05.388730 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.388697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:01:05.763837 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.763749    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"]
Sep 04 23:01:05.772966 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.772928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:05.773461 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.773427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"]
Sep 04 23:01:05.775780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.775722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ncjwg\""
Sep 04 23:01:05.776137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.776053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:01:05.891064 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.891016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7\" (UID: \"54dfb972-ecbc-4830-aeff-c64861808415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:05.897561 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.897535    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:01:05.991844 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.991805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7\" (UID: \"54dfb972-ecbc-4830-aeff-c64861808415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:05.992419 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:05.992399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7\" (UID: \"54dfb972-ecbc-4830-aeff-c64861808415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:06.086886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:06.086840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:06.357446 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:06.357418    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"]
Sep 04 23:01:06.946144 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:06.946107    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7" event={"ID":"54dfb972-ecbc-4830-aeff-c64861808415","Type":"ContainerStarted","Data":"cd150de05cac6332f6e0f15e99a1c5251013e766993afd0e0f5879e9991c3980"}
Sep 04 23:01:08.956097 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.956062    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerID="3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f" exitCode=0
Sep 04 23:01:08.956551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.956153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerDied","Data":"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"}
Sep 04 23:01:08.956662 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.956643    2577 scope.go:117] "RemoveContainer" containerID="3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"
Sep 04 23:01:08.957815 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.957787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7" event={"ID":"54dfb972-ecbc-4830-aeff-c64861808415","Type":"ContainerStarted","Data":"e0490ab3e75b97aad5e23bffc48fd4e42ca50c7aaa930cf5582a54128f46cae6"}
Sep 04 23:01:08.957909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.957843    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:08.957953 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.957913    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:08.957987 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.957951    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:08.994787 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:08.994709    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7" podStartSLOduration=1.866655712 podStartE2EDuration="3.994692836s" totalImagesPullingTime="2.128037124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:08.99393278 +0000 UTC m=+953.165007391" watchObservedRunningTime="2026-09-04 23:01:08.994692836 +0000 UTC m=+953.165767447"
Sep 04 23:01:09.965986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:09.965954    2577 generic.go:356] "Generic (PLEG): container finished" podID="54dfb972-ecbc-4830-aeff-c64861808415" containerID="e0490ab3e75b97aad5e23bffc48fd4e42ca50c7aaa930cf5582a54128f46cae6" exitCode=0
Sep 04 23:01:09.966415 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:09.966047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7" event={"ID":"54dfb972-ecbc-4830-aeff-c64861808415","Type":"ContainerDied","Data":"e0490ab3e75b97aad5e23bffc48fd4e42ca50c7aaa930cf5582a54128f46cae6"}
Sep 04 23:01:09.967991 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:09.967971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerStarted","Data":"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"}
Sep 04 23:01:10.422180 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.422133    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:01:10.422423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.422396    2577 patch_prober.go:28] interesting pod/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 04 23:01:10.422476 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.422456    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 04 23:01:10.790482 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.790396    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:10.792642 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.792622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.795277 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.795257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-d5be7135-kserve-self-signed-certs\""
Sep 04 23:01:10.802940 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.802916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:10.943458 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.943410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.943825 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.943791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.943988 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.943970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwlsp\" (UniqueName: \"kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.944160 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.944131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.944331 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.944306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:10.944542 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:10.944526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwlsp\" (UniqueName: \"kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.050883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.049912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.051614 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.051414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.051756 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.051713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.063162 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.063106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.066171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.066112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwlsp\" (UniqueName: \"kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.067431 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.067404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm\") pod \"e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") " pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.102282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.102184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:11.116758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.116708    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:11.253761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.252147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca\") pod \"54dfb972-ecbc-4830-aeff-c64861808415\" (UID: \"54dfb972-ecbc-4830-aeff-c64861808415\") "
Sep 04 23:01:11.253761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.252822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54dfb972-ecbc-4830-aeff-c64861808415" (UID: "54dfb972-ecbc-4830-aeff-c64861808415"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:01:11.353761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.353271    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54dfb972-ecbc-4830-aeff-c64861808415-maas-api-service-ca\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:11.404159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.404119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:11.406535 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:01:11.406507    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb89a2c77_5dc4_41bf_9893_6071e5ad1baf.slice/crio-d76c2c1eb6a96346a3556b952184b1c0aa9bf7148bb00894351e9feba0a17ee3 WatchSource:0}: Error finding container d76c2c1eb6a96346a3556b952184b1c0aa9bf7148bb00894351e9feba0a17ee3: Status 404 returned error can't find the container with id d76c2c1eb6a96346a3556b952184b1c0aa9bf7148bb00894351e9feba0a17ee3
Sep 04 23:01:11.422618 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.422589    2577 patch_prober.go:28] interesting pod/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 04 23:01:11.422747 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.422653    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 04 23:01:11.978102 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.978072    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"
Sep 04 23:01:11.978304 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.978071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7" event={"ID":"54dfb972-ecbc-4830-aeff-c64861808415","Type":"ContainerDied","Data":"cd150de05cac6332f6e0f15e99a1c5251013e766993afd0e0f5879e9991c3980"}
Sep 04 23:01:11.978304 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.978208    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd150de05cac6332f6e0f15e99a1c5251013e766993afd0e0f5879e9991c3980"
Sep 04 23:01:11.979634 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.979600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" event={"ID":"b89a2c77-5dc4-41bf-9893-6071e5ad1baf","Type":"ContainerStarted","Data":"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"}
Sep 04 23:01:11.979763 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.979641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" event={"ID":"b89a2c77-5dc4-41bf-9893-6071e5ad1baf","Type":"ContainerStarted","Data":"d76c2c1eb6a96346a3556b952184b1c0aa9bf7148bb00894351e9feba0a17ee3"}
Sep 04 23:01:11.996749 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:11.996692    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" podStartSLOduration=1.996678671 podStartE2EDuration="1.996678671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:11.99514642 +0000 UTC m=+956.166221034" watchObservedRunningTime="2026-09-04 23:01:11.996678671 +0000 UTC m=+956.167753282"
Sep 04 23:01:12.421954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:12.421915    2577 patch_prober.go:28] interesting pod/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 04 23:01:12.422348 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:12.421980    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 04 23:01:13.422360 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:13.422320    2577 patch_prober.go:28] interesting pod/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 04 23:01:13.422767 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:13.422393    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 04 23:01:14.422312 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:14.422274    2577 patch_prober.go:28] interesting pod/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Sep 04 23:01:14.422492 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:14.422330    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Sep 04 23:01:15.469723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:15.469679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:01:15.470214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:15.470188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:01:15.471842 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:15.471819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:01:21.102598 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:21.102546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:21.103125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:21.102621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:21.115007 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:21.114980    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:21.606851 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:21.606811    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:01:22.035724 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:22.035641    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:22.667946 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:22.667907    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"]
Sep 04 23:01:22.669835 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:22.669814    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-921d8a5d-q8mj7"]
Sep 04 23:01:24.324899 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:24.324869    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54dfb972-ecbc-4830-aeff-c64861808415" path="/var/lib/kubelet/pods/54dfb972-ecbc-4830-aeff-c64861808415/volumes"
Sep 04 23:01:27.339955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.339921    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:27.340406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.340225    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dfb972-ecbc-4830-aeff-c64861808415" containerName="revoke-keys"
Sep 04 23:01:27.340406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.340235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dfb972-ecbc-4830-aeff-c64861808415" containerName="revoke-keys"
Sep 04 23:01:27.340406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.340289    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dfb972-ecbc-4830-aeff-c64861808415" containerName="revoke-keys"
Sep 04 23:01:27.348515 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.348493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.351218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.351197    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4cc4c1dc\""
Sep 04 23:01:27.351640 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.351614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4cc4c1dc-dockercfg-bbhj8\""
Sep 04 23:01:27.355380 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.355354    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:27.380508 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.380476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.380644 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.380553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw2t\" (UniqueName: \"kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.405750 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.405713    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:27.409411 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.409394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.417380 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.417346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:27.481861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.481821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.481861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.481864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw2t\" (UniqueName: \"kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.482078 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.481950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf4d\" (UniqueName: \"kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.482078 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.482005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.482688 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.482671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.490713 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.490684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw2t\" (UniqueName: \"kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t\") pod \"payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") " pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.582558 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.582513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.582798 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.582598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf4d\" (UniqueName: \"kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.583278 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.583257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.590833 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.590782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf4d\" (UniqueName: \"kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d\") pod \"payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.663721 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.663681    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:27.720414 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.720340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:27.934550 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.934482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:27.937142 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:01:27.937111    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d42d48_9883_493a_99b8_e0c21231686f.slice/crio-e2c3aaa57dd90417503972f781c332a2b23973564c62665a5a28149d2e057c94 WatchSource:0}: Error finding container e2c3aaa57dd90417503972f781c332a2b23973564c62665a5a28149d2e057c94: Status 404 returned error can't find the container with id e2c3aaa57dd90417503972f781c332a2b23973564c62665a5a28149d2e057c94
Sep 04 23:01:27.994497 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:27.994464    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:27.997175 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:01:27.997150    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bb90ec_b59c_4106_9a36_98a82e60bea5.slice/crio-d530e0192aa281bf4d7c1de93be27be5e41dd2fe7a8b5322243d68c263b9b997 WatchSource:0}: Error finding container d530e0192aa281bf4d7c1de93be27be5e41dd2fe7a8b5322243d68c263b9b997: Status 404 returned error can't find the container with id d530e0192aa281bf4d7c1de93be27be5e41dd2fe7a8b5322243d68c263b9b997
Sep 04 23:01:28.052699 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:28.052652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" event={"ID":"d2d42d48-9883-493a-99b8-e0c21231686f","Type":"ContainerStarted","Data":"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"}
Sep 04 23:01:28.052907 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:28.052729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" event={"ID":"d2d42d48-9883-493a-99b8-e0c21231686f","Type":"ContainerStarted","Data":"e2c3aaa57dd90417503972f781c332a2b23973564c62665a5a28149d2e057c94"}
Sep 04 23:01:28.052907 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:28.052858    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:28.054149 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:28.054117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" event={"ID":"51bb90ec-b59c-4106-9a36-98a82e60bea5","Type":"ContainerStarted","Data":"d530e0192aa281bf4d7c1de93be27be5e41dd2fe7a8b5322243d68c263b9b997"}
Sep 04 23:01:29.059925 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:29.059891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" event={"ID":"51bb90ec-b59c-4106-9a36-98a82e60bea5","Type":"ContainerStarted","Data":"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"}
Sep 04 23:01:29.060312 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:29.060134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:29.075974 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:29.075907    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" podStartSLOduration=2.075885154 podStartE2EDuration="2.075885154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:28.069517605 +0000 UTC m=+972.240592229" watchObservedRunningTime="2026-09-04 23:01:29.075885154 +0000 UTC m=+973.246959768"
Sep 04 23:01:29.077224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:29.077188    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" podStartSLOduration=2.077177522 podStartE2EDuration="2.077177522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:29.075560447 +0000 UTC m=+973.246635060" watchObservedRunningTime="2026-09-04 23:01:29.077177522 +0000 UTC m=+973.248252128"
Sep 04 23:01:37.478064 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.478023    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:01:37.481668 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.481648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.485263 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.485237    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63d928\"/\"kube-root-ca.crt\""
Sep 04 23:01:37.485528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.485497    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-63d928\"/\"openshift-service-ca.crt\""
Sep 04 23:01:37.485528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.485518    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63d928\"/\"autores-model-63d928-kserve-self-signed-certs\""
Sep 04 23:01:37.485710 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.485577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-63d928\"/\"default-dockercfg-d79nd\""
Sep 04 23:01:37.490138 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.490109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:01:37.569806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.569769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.570003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.569815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxv9j\" (UniqueName: \"kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.570003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.569940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.570003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.569981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.570177 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.570021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.570177 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.570096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671549 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671549 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671549 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671549 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxv9j\" (UniqueName: \"kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.671928 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.671905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.673633 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.673608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.673931 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.673915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.679215 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.679190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxv9j\" (UniqueName: \"kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j\") pod \"autores-model-63d928-kserve-6f8d67f4f5-zrgv9\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") " pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:37.793439 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:37.793354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:38.062399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:38.062374    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:01:38.064329 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:01:38.064302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12393d20_75dd_495d_b7ee_3820db640716.slice/crio-8199b4ea620d70676dfd33f6cd0677418e377eac44b9142938382973be90e14e WatchSource:0}: Error finding container 8199b4ea620d70676dfd33f6cd0677418e377eac44b9142938382973be90e14e: Status 404 returned error can't find the container with id 8199b4ea620d70676dfd33f6cd0677418e377eac44b9142938382973be90e14e
Sep 04 23:01:38.104626 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:38.104597    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" event={"ID":"12393d20-75dd-495d-b7ee-3820db640716","Type":"ContainerStarted","Data":"8199b4ea620d70676dfd33f6cd0677418e377eac44b9142938382973be90e14e"}
Sep 04 23:01:39.062445 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:39.062415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:39.115512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:39.115483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" event={"ID":"12393d20-75dd-495d-b7ee-3820db640716","Type":"ContainerStarted","Data":"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"}
Sep 04 23:01:39.134817 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:39.134773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" podStartSLOduration=2.134732066 podStartE2EDuration="2.134732066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:39.133382219 +0000 UTC m=+983.304456831" watchObservedRunningTime="2026-09-04 23:01:39.134732066 +0000 UTC m=+983.305806678"
Sep 04 23:01:40.066685 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:40.066653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:41.872254 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:41.872214    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:43.598343 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:43.598298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:01:47.573792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:47.573758    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:01:47.794129 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:47.794098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:01:48.567341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:48.567297    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:01:56.893802 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:56.893759    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" podUID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" containerName="main" containerID="cri-o://f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31" gracePeriod=45
Sep 04 23:01:57.152117 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.152050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170142    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwlsp\" (UniqueName: \"kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170226    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm\") pod \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\" (UID: \"b89a2c77-5dc4-41bf-9893-6071e5ad1baf\") "
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:01:57.170761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170552    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home" (OuterVolumeSpecName: "home") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:01:57.171265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170788    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.171265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170810    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.171265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.170840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:01:57.172795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.172765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp" (OuterVolumeSpecName: "kube-api-access-lwlsp") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "kube-api-access-lwlsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:01:57.172795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.172765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:01:57.173207 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.173188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm" (OuterVolumeSpecName: "dshm") pod "b89a2c77-5dc4-41bf-9893-6071e5ad1baf" (UID: "b89a2c77-5dc4-41bf-9893-6071e5ad1baf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:01:57.206382 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.206345    2577 generic.go:356] "Generic (PLEG): container finished" podID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" containerID="f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31" exitCode=0
Sep 04 23:01:57.206382 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.206383    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"
Sep 04 23:01:57.206600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.206416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" event={"ID":"b89a2c77-5dc4-41bf-9893-6071e5ad1baf","Type":"ContainerDied","Data":"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"}
Sep 04 23:01:57.206600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.206450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb" event={"ID":"b89a2c77-5dc4-41bf-9893-6071e5ad1baf","Type":"ContainerDied","Data":"d76c2c1eb6a96346a3556b952184b1c0aa9bf7148bb00894351e9feba0a17ee3"}
Sep 04 23:01:57.206600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.206466    2577 scope.go:117] "RemoveContainer" containerID="f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"
Sep 04 23:01:57.215275 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.215259    2577 scope.go:117] "RemoveContainer" containerID="f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"
Sep 04 23:01:57.215557 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:01:57.215540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31\": container with ID starting with f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31 not found: ID does not exist" containerID="f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"
Sep 04 23:01:57.215601 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.215565    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31"} err="failed to get container status \"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31\": rpc error: code = NotFound desc = could not find container \"f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31\": container with ID starting with f13fde40432e01d78fed37631fa7073295ffc314ea504afba4a7ce84706edf31 not found: ID does not exist"
Sep 04 23:01:57.226990 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.226964    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:57.229722 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.229702    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-d5be7135-kserve-6f446484f7-xnbwb"]
Sep 04 23:01:57.271773 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.271725    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwlsp\" (UniqueName: \"kubernetes.io/projected/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-kube-api-access-lwlsp\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.271915 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.271777    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.271915 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.271795    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.271915 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.271806    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b89a2c77-5dc4-41bf-9893-6071e5ad1baf-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.420410 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.420322    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:57.420590 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.420545    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" podUID="d2d42d48-9883-493a-99b8-e0c21231686f" containerName="payload-processing" containerID="cri-o://65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239" gracePeriod=30
Sep 04 23:01:57.428324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.428294    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:57.428494 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.428473    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" podUID="51bb90ec-b59c-4106-9a36-98a82e60bea5" containerName="payload-pre-processing" containerID="cri-o://07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454" gracePeriod=30
Sep 04 23:01:57.685305 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.685281    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:57.693983 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.693962    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:57.776130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume\") pod \"d2d42d48-9883-493a-99b8-e0c21231686f\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") "
Sep 04 23:01:57.776271 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnf4d\" (UniqueName: \"kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d\") pod \"51bb90ec-b59c-4106-9a36-98a82e60bea5\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") "
Sep 04 23:01:57.776271 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776196    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlw2t\" (UniqueName: \"kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t\") pod \"d2d42d48-9883-493a-99b8-e0c21231686f\" (UID: \"d2d42d48-9883-493a-99b8-e0c21231686f\") "
Sep 04 23:01:57.776271 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume\") pod \"51bb90ec-b59c-4106-9a36-98a82e60bea5\" (UID: \"51bb90ec-b59c-4106-9a36-98a82e60bea5\") "
Sep 04 23:01:57.776591 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2d42d48-9883-493a-99b8-e0c21231686f" (UID: "d2d42d48-9883-493a-99b8-e0c21231686f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:01:57.776651 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.776596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51bb90ec-b59c-4106-9a36-98a82e60bea5" (UID: "51bb90ec-b59c-4106-9a36-98a82e60bea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:01:57.778307 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.778285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t" (OuterVolumeSpecName: "kube-api-access-hlw2t") pod "d2d42d48-9883-493a-99b8-e0c21231686f" (UID: "d2d42d48-9883-493a-99b8-e0c21231686f"). InnerVolumeSpecName "kube-api-access-hlw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:01:57.778362 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.778306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d" (OuterVolumeSpecName: "kube-api-access-lnf4d") pod "51bb90ec-b59c-4106-9a36-98a82e60bea5" (UID: "51bb90ec-b59c-4106-9a36-98a82e60bea5"). InnerVolumeSpecName "kube-api-access-lnf4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:01:57.877753 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.877716    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d42d48-9883-493a-99b8-e0c21231686f-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.877753 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.877756    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnf4d\" (UniqueName: \"kubernetes.io/projected/51bb90ec-b59c-4106-9a36-98a82e60bea5-kube-api-access-lnf4d\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.877954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.877770    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlw2t\" (UniqueName: \"kubernetes.io/projected/d2d42d48-9883-493a-99b8-e0c21231686f-kube-api-access-hlw2t\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:57.877954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:57.877782    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bb90ec-b59c-4106-9a36-98a82e60bea5-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:01:58.212880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.212843    2577 generic.go:356] "Generic (PLEG): container finished" podID="51bb90ec-b59c-4106-9a36-98a82e60bea5" containerID="07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454" exitCode=0
Sep 04 23:01:58.213425 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.212894    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"
Sep 04 23:01:58.213425 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.212934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" event={"ID":"51bb90ec-b59c-4106-9a36-98a82e60bea5","Type":"ContainerDied","Data":"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"}
Sep 04 23:01:58.213425 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.212992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p" event={"ID":"51bb90ec-b59c-4106-9a36-98a82e60bea5","Type":"ContainerDied","Data":"d530e0192aa281bf4d7c1de93be27be5e41dd2fe7a8b5322243d68c263b9b997"}
Sep 04 23:01:58.213425 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.213011    2577 scope.go:117] "RemoveContainer" containerID="07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"
Sep 04 23:01:58.214457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.214313    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2d42d48-9883-493a-99b8-e0c21231686f" containerID="65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239" exitCode=0
Sep 04 23:01:58.214457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.214355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" event={"ID":"d2d42d48-9883-493a-99b8-e0c21231686f","Type":"ContainerDied","Data":"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"}
Sep 04 23:01:58.214457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.214369    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"
Sep 04 23:01:58.214457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.214385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss" event={"ID":"d2d42d48-9883-493a-99b8-e0c21231686f","Type":"ContainerDied","Data":"e2c3aaa57dd90417503972f781c332a2b23973564c62665a5a28149d2e057c94"}
Sep 04 23:01:58.222854 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.222837    2577 scope.go:117] "RemoveContainer" containerID="07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"
Sep 04 23:01:58.223165 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:01:58.223146    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454\": container with ID starting with 07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454 not found: ID does not exist" containerID="07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"
Sep 04 23:01:58.223256 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.223170    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454"} err="failed to get container status \"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454\": rpc error: code = NotFound desc = could not find container \"07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454\": container with ID starting with 07871e0855bc2bcb5b47c550e852efd02943efdb27dc2a1e6f0b781adfd5d454 not found: ID does not exist"
Sep 04 23:01:58.223256 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.223186    2577 scope.go:117] "RemoveContainer" containerID="65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"
Sep 04 23:01:58.231110 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.231093    2577 scope.go:117] "RemoveContainer" containerID="65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"
Sep 04 23:01:58.231374 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:01:58.231353    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239\": container with ID starting with 65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239 not found: ID does not exist" containerID="65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"
Sep 04 23:01:58.231433 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.231384    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239"} err="failed to get container status \"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239\": rpc error: code = NotFound desc = could not find container \"65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239\": container with ID starting with 65fdcabb4a293f0c1e665ba39049343635425e83c78d50aa3d5907ac89889239 not found: ID does not exist"
Sep 04 23:01:58.234877 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.234853    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:58.237009 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.236990    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4cc4c1dc-5887774b66-4928p"]
Sep 04 23:01:58.246962 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.246941    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:58.250982 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.250960    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4cc4c1dc-7569bd7c4-flfss"]
Sep 04 23:01:58.325273 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.325246    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51bb90ec-b59c-4106-9a36-98a82e60bea5" path="/var/lib/kubelet/pods/51bb90ec-b59c-4106-9a36-98a82e60bea5/volumes"
Sep 04 23:01:58.325631 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.325619    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" path="/var/lib/kubelet/pods/b89a2c77-5dc4-41bf-9893-6071e5ad1baf/volumes"
Sep 04 23:01:58.325969 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.325957    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d42d48-9883-493a-99b8-e0c21231686f" path="/var/lib/kubelet/pods/d2d42d48-9883-493a-99b8-e0c21231686f/volumes"
Sep 04 23:01:58.471292 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471201    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:01:58.471528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471513    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bb90ec-b59c-4106-9a36-98a82e60bea5" containerName="payload-pre-processing"
Sep 04 23:01:58.471528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471528    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bb90ec-b59c-4106-9a36-98a82e60bea5" containerName="payload-pre-processing"
Sep 04 23:01:58.471655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471545    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" containerName="main"
Sep 04 23:01:58.471655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471552    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" containerName="main"
Sep 04 23:01:58.471655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471559    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d42d48-9883-493a-99b8-e0c21231686f" containerName="payload-processing"
Sep 04 23:01:58.471655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471565    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d42d48-9883-493a-99b8-e0c21231686f" containerName="payload-processing"
Sep 04 23:01:58.471857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b89a2c77-5dc4-41bf-9893-6071e5ad1baf" containerName="main"
Sep 04 23:01:58.471857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bb90ec-b59c-4106-9a36-98a82e60bea5" containerName="payload-pre-processing"
Sep 04 23:01:58.471857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.471803    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d42d48-9883-493a-99b8-e0c21231686f" containerName="payload-processing"
Sep 04 23:01:58.476216 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.476192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.478753 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.478713    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e8480dce-openshift-default-dockercfg-h5c5n\""
Sep 04 23:01:58.482394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482614    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9b5\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.482865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.482730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.487819 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.487793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:01:58.584486 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.584451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9b5\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.584787 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.584766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.584922 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.584908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585217 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585320 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585426 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585526 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.585635 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.585622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.586216 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.586192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.586583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.586563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.587302 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.587282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.589778 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.589752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.589927 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.589799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.590115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.590093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.592205 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.592173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.593138 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.593110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9b5\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.597909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.597891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token\") pod \"e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") " pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:58.786760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:58.786642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:01:59.065799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.065757    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:01:59.068408 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:01:59.068379    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d236955_e765_4539_a04e_7da81e9b36c1.slice/crio-16e85029f83dd044a1833e323d278906dec48a1b6f9e07565fa1589abee69a2c WatchSource:0}: Error finding container 16e85029f83dd044a1833e323d278906dec48a1b6f9e07565fa1589abee69a2c: Status 404 returned error can't find the container with id 16e85029f83dd044a1833e323d278906dec48a1b6f9e07565fa1589abee69a2c
Sep 04 23:01:59.070626 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.070583    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:59.070689 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.070675    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:59.070754 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.070719    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:01:59.221228 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.221194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerStarted","Data":"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"}
Sep 04 23:01:59.221655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.221251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerStarted","Data":"16e85029f83dd044a1833e323d278906dec48a1b6f9e07565fa1589abee69a2c"}
Sep 04 23:01:59.241790 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.241697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podStartSLOduration=1.241676067 podStartE2EDuration="1.241676067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:01:59.239337548 +0000 UTC m=+1003.410412217" watchObservedRunningTime="2026-09-04 23:01:59.241676067 +0000 UTC m=+1003.412750681"
Sep 04 23:01:59.787792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:01:59.787762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:00.792466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:00.792437    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:01.232644 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:01.232608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:01.233555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:01.233537    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:02.302929 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.302890    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:02:02.593586 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.593490    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" podUID="12393d20-75dd-495d-b7ee-3820db640716" containerName="main" containerID="cri-o://eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9" gracePeriod=45
Sep 04 23:02:02.658750 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.658710    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:02.661989 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.661967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.664389 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.664371    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e8480dce\""
Sep 04 23:02:02.664459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.664417    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e8480dce-dockercfg-nrthq\""
Sep 04 23:02:02.673362 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.673340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:02.723446 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.723416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.723594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.723527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bdsh\" (UniqueName: \"kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.825058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.825015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.825223 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.825143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bdsh\" (UniqueName: \"kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.825817 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.825795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.833305 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.833277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bdsh\" (UniqueName: \"kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh\") pod \"payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:02.850689 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.850623    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:02:02.926103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxv9j\" (UniqueName: \"kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926165    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir\") pod \"12393d20-75dd-495d-b7ee-3820db640716\" (UID: \"12393d20-75dd-495d-b7ee-3820db640716\") "
Sep 04 23:02:02.926532 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home" (OuterVolumeSpecName: "home") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:02.926532 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926431    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache" (OuterVolumeSpecName: "model-cache") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:02.926640 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926515    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:02.926640 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926574    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:02.926640 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.926582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:02.928478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.928449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm" (OuterVolumeSpecName: "dshm") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:02.928560 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.928461    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:02:02.928600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.928570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j" (OuterVolumeSpecName: "kube-api-access-bxv9j") pod "12393d20-75dd-495d-b7ee-3820db640716" (UID: "12393d20-75dd-495d-b7ee-3820db640716"). InnerVolumeSpecName "kube-api-access-bxv9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:02.971995 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:02.971953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:03.028229 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.028138    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxv9j\" (UniqueName: \"kubernetes.io/projected/12393d20-75dd-495d-b7ee-3820db640716-kube-api-access-bxv9j\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:03.028229 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.028173    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12393d20-75dd-495d-b7ee-3820db640716-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:03.028229 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.028191    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:03.028229 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.028205    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12393d20-75dd-495d-b7ee-3820db640716-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:03.242777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.242678    2577 generic.go:356] "Generic (PLEG): container finished" podID="12393d20-75dd-495d-b7ee-3820db640716" containerID="eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9" exitCode=0
Sep 04 23:02:03.242777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.242722    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"
Sep 04 23:02:03.242777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.242764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" event={"ID":"12393d20-75dd-495d-b7ee-3820db640716","Type":"ContainerDied","Data":"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"}
Sep 04 23:02:03.242991 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.242814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9" event={"ID":"12393d20-75dd-495d-b7ee-3820db640716","Type":"ContainerDied","Data":"8199b4ea620d70676dfd33f6cd0677418e377eac44b9142938382973be90e14e"}
Sep 04 23:02:03.242991 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.242836    2577 scope.go:117] "RemoveContainer" containerID="eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"
Sep 04 23:02:03.250311 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.250292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:03.252165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.252149    2577 scope.go:117] "RemoveContainer" containerID="eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"
Sep 04 23:02:03.252451 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:03.252427    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9\": container with ID starting with eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9 not found: ID does not exist" containerID="eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"
Sep 04 23:02:03.252498 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.252463    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9"} err="failed to get container status \"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9\": rpc error: code = NotFound desc = could not find container \"eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9\": container with ID starting with eb8447be56bce4c4ced26587d4d0c2f1f2deae1c1ef65f1bbd24011de2630ee9 not found: ID does not exist"
Sep 04 23:02:03.253293 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:03.253271    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4bdb92_8f36_4456_899b_abac40878869.slice/crio-18db517128b31ec47edb72d46cc1cd8486058d2404b86bf81039c959269899c4 WatchSource:0}: Error finding container 18db517128b31ec47edb72d46cc1cd8486058d2404b86bf81039c959269899c4: Status 404 returned error can't find the container with id 18db517128b31ec47edb72d46cc1cd8486058d2404b86bf81039c959269899c4
Sep 04 23:02:03.267783 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.267753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:02:03.267961 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.267949    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-63d928/autores-model-63d928-kserve-6f8d67f4f5-zrgv9"]
Sep 04 23:02:03.586568 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.586522    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" podUID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" containerName="main" containerID="cri-o://04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9" gracePeriod=45
Sep 04 23:02:03.827788 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.827758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:02:03.934971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.934885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.934971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.934930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skj4j\" (UniqueName: \"kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.934971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.934957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.934971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.934974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.935297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.935297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm\") pod \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\" (UID: \"954dc02b-d0d0-401e-8a46-84d35cf1cf29\") "
Sep 04 23:02:03.935297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935202    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache" (OuterVolumeSpecName: "model-cache") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:03.935411 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:03.935411 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home" (OuterVolumeSpecName: "home") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:03.935411 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.935322    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:03.937057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.937032    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm" (OuterVolumeSpecName: "dshm") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:03.937533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.937512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:02:03.937533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:03.937518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j" (OuterVolumeSpecName: "kube-api-access-skj4j") pod "954dc02b-d0d0-401e-8a46-84d35cf1cf29" (UID: "954dc02b-d0d0-401e-8a46-84d35cf1cf29"). InnerVolumeSpecName "kube-api-access-skj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:04.036159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.036116    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:04.036159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.036150    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:04.036159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.036162    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skj4j\" (UniqueName: \"kubernetes.io/projected/954dc02b-d0d0-401e-8a46-84d35cf1cf29-kube-api-access-skj4j\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:04.036394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.036177    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:04.036394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.036190    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954dc02b-d0d0-401e-8a46-84d35cf1cf29-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:04.250963 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.250876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" event={"ID":"cd4bdb92-8f36-4456-899b-abac40878869","Type":"ContainerStarted","Data":"a2f195e8559b9da271655e2586948cb8e1485661224b68cb5b54d3259faca747"}
Sep 04 23:02:04.251120 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.250985    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:04.251120 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.251016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" event={"ID":"cd4bdb92-8f36-4456-899b-abac40878869","Type":"ContainerStarted","Data":"18db517128b31ec47edb72d46cc1cd8486058d2404b86bf81039c959269899c4"}
Sep 04 23:02:04.252174 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.252149    2577 generic.go:356] "Generic (PLEG): container finished" podID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" containerID="04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9" exitCode=0
Sep 04 23:02:04.252236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.252184    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"
Sep 04 23:02:04.252236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.252195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" event={"ID":"954dc02b-d0d0-401e-8a46-84d35cf1cf29","Type":"ContainerDied","Data":"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"}
Sep 04 23:02:04.252236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.252217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp" event={"ID":"954dc02b-d0d0-401e-8a46-84d35cf1cf29","Type":"ContainerDied","Data":"b05584afab8484aefcac3cffc133b143012777b69aa958ee7a350189cf344e85"}
Sep 04 23:02:04.252330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.252237    2577 scope.go:117] "RemoveContainer" containerID="04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"
Sep 04 23:02:04.261527 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.261500    2577 scope.go:117] "RemoveContainer" containerID="04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"
Sep 04 23:02:04.261903 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:04.261876    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9\": container with ID starting with 04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9 not found: ID does not exist" containerID="04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"
Sep 04 23:02:04.262005 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.261920    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9"} err="failed to get container status \"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9\": rpc error: code = NotFound desc = could not find container \"04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9\": container with ID starting with 04efa966704d8c589806855571d2b939e58f7b0bdb9c11600effc4694e6969b9 not found: ID does not exist"
Sep 04 23:02:04.268373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.268334    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" podStartSLOduration=2.268321119 podStartE2EDuration="2.268321119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:04.266644224 +0000 UTC m=+1008.437718840" watchObservedRunningTime="2026-09-04 23:02:04.268321119 +0000 UTC m=+1008.439395729"
Sep 04 23:02:04.283594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.283570    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:02:04.286491 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.286472    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-d5be7135-kserve-7f6b9fdb5f-c5tvp"]
Sep 04 23:02:04.325803 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.325761    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12393d20-75dd-495d-b7ee-3820db640716" path="/var/lib/kubelet/pods/12393d20-75dd-495d-b7ee-3820db640716/volumes"
Sep 04 23:02:04.326211 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:04.326197    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" path="/var/lib/kubelet/pods/954dc02b-d0d0-401e-8a46-84d35cf1cf29/volumes"
Sep 04 23:02:05.692473 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:05.692439    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:02:05.692887 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:05.692655    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy" containerID="cri-o://d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191" gracePeriod=30
Sep 04 23:02:10.952553 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.952526    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:02:10.992447 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992533    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gff6\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy\") pod \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\" (UID: \"31d629a3-edc0-410d-aa4a-44cb056bdfb5\") "
Sep 04 23:02:10.992892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:10.992892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.992859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:10.993135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.993073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:10.993135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.993118    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:10.993234 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.993142    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-data\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:10.993359 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.993311    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:10.993493 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.993371    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:02:10.995050 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.995023    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:10.995544 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.995514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:10.995909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.995886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:02:10.996395 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:10.996373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6" (OuterVolumeSpecName: "kube-api-access-5gff6") pod "31d629a3-edc0-410d-aa4a-44cb056bdfb5" (UID: "31d629a3-edc0-410d-aa4a-44cb056bdfb5"). InnerVolumeSpecName "kube-api-access-5gff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:11.094321 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094288    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-podinfo\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094321 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094320    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istiod-ca-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094332    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-workload-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094343    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-credential-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094357    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gff6\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-kube-api-access-5gff6\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094368    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-envoy\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.094516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.094378    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31d629a3-edc0-410d-aa4a-44cb056bdfb5-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:11.153373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153340    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:02:11.153648 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153636    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.153695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.153695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153665    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12393d20-75dd-495d-b7ee-3820db640716" containerName="main"
Sep 04 23:02:11.153695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153670    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12393d20-75dd-495d-b7ee-3820db640716" containerName="main"
Sep 04 23:02:11.153695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153679    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" containerName="main"
Sep 04 23:02:11.153695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153684    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" containerName="main"
Sep 04 23:02:11.153889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153747    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.153889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153757    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.153889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153764    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12393d20-75dd-495d-b7ee-3820db640716" containerName="main"
Sep 04 23:02:11.153889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.153771    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="954dc02b-d0d0-401e-8a46-84d35cf1cf29" containerName="main"
Sep 04 23:02:11.156808 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.156781    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.159253 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.159229    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-72563d\""
Sep 04 23:02:11.168500 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.168473    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:02:11.175040 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.175016    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:02:11.175330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.175317    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.175330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.175330    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerName="istio-proxy"
Sep 04 23:02:11.178109 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.178092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.181039 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.181022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-72563d-dockercfg-9gvm7\""
Sep 04 23:02:11.188450 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.188430    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:02:11.195150 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.195128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gxhm\" (UniqueName: \"kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.195278 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.195214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.288434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.288345    2577 generic.go:356] "Generic (PLEG): container finished" podID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" containerID="d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191" exitCode=0
Sep 04 23:02:11.288434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.288387    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"
Sep 04 23:02:11.288434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.288397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerDied","Data":"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"}
Sep 04 23:02:11.288715 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.288447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79" event={"ID":"31d629a3-edc0-410d-aa4a-44cb056bdfb5","Type":"ContainerDied","Data":"81a20c23bc6fdf32b6589a0cabcc126e7fe1bb4a0e6f8d73bdd45df1ae0cbcc5"}
Sep 04 23:02:11.288715 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.288472    2577 scope.go:117] "RemoveContainer" containerID="d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"
Sep 04 23:02:11.296036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.295995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.296164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.296039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvrl\" (UniqueName: \"kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.296164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.296154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.296276 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.296213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gxhm\" (UniqueName: \"kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.297057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.297033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.300527 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.300506    2577 scope.go:117] "RemoveContainer" containerID="e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"
Sep 04 23:02:11.307553 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.307530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gxhm\" (UniqueName: \"kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm\") pod \"payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.310536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.310511    2577 scope.go:117] "RemoveContainer" containerID="d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"
Sep 04 23:02:11.310882 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:11.310859    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191\": container with ID starting with d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191 not found: ID does not exist" containerID="d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"
Sep 04 23:02:11.310962 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.310893    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191"} err="failed to get container status \"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191\": rpc error: code = NotFound desc = could not find container \"d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191\": container with ID starting with d578e9b34c44fb47a06752eed6c9227082017e7e8b410101c036768d77b7a191 not found: ID does not exist"
Sep 04 23:02:11.310962 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.310917    2577 scope.go:117] "RemoveContainer" containerID="e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"
Sep 04 23:02:11.311278 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:11.311251    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d\": container with ID starting with e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d not found: ID does not exist" containerID="e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"
Sep 04 23:02:11.311370 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.311284    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d"} err="failed to get container status \"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d\": rpc error: code = NotFound desc = could not find container \"e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d\": container with ID starting with e27300f5c4a90534f730fcaf3aedbcbfc850f1a388fa80e39c330fcb01e3cd5d not found: ID does not exist"
Sep 04 23:02:11.315645 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.315588    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:02:11.318132 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.318108    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-63d928-openshift-default-6565cd9666-w7r79"]
Sep 04 23:02:11.397009 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.396975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.397009 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.397012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvrl\" (UniqueName: \"kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.397668 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.397647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.404823 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.404800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvrl\" (UniqueName: \"kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl\") pod \"payload-processing-e2e-explicit-72563d-556c84c889-vx8b2\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") " pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.466544 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.466506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:11.489713 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.489682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:11.741761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.741635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:02:11.751813 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:11.745413    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d457d9_728f_47e3_90df_ddbc9efe16d7.slice/crio-d163e250de101101868669f4882203cca14edc2cff6423cf3dc7ab8d9d4befb2 WatchSource:0}: Error finding container d163e250de101101868669f4882203cca14edc2cff6423cf3dc7ab8d9d4befb2: Status 404 returned error can't find the container with id d163e250de101101868669f4882203cca14edc2cff6423cf3dc7ab8d9d4befb2
Sep 04 23:02:11.777005 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:11.776980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:02:11.779950 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:11.779926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6db88d_f65d_4ad2_a218_0925b6aaef06.slice/crio-e3d2411fec1b9ea41665965e8efafc20b1e2b122fb590eecb07871bc589cef4d WatchSource:0}: Error finding container e3d2411fec1b9ea41665965e8efafc20b1e2b122fb590eecb07871bc589cef4d: Status 404 returned error can't find the container with id e3d2411fec1b9ea41665965e8efafc20b1e2b122fb590eecb07871bc589cef4d
Sep 04 23:02:12.295883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.295840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" event={"ID":"5d6db88d-f65d-4ad2-a218-0925b6aaef06","Type":"ContainerStarted","Data":"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"}
Sep 04 23:02:12.296341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.295941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:12.296341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.295971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" event={"ID":"5d6db88d-f65d-4ad2-a218-0925b6aaef06","Type":"ContainerStarted","Data":"e3d2411fec1b9ea41665965e8efafc20b1e2b122fb590eecb07871bc589cef4d"}
Sep 04 23:02:12.297442 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.297414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" event={"ID":"29d457d9-728f-47e3-90df-ddbc9efe16d7","Type":"ContainerStarted","Data":"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"}
Sep 04 23:02:12.297578 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.297457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" event={"ID":"29d457d9-728f-47e3-90df-ddbc9efe16d7","Type":"ContainerStarted","Data":"d163e250de101101868669f4882203cca14edc2cff6423cf3dc7ab8d9d4befb2"}
Sep 04 23:02:12.297578 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.297518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:12.314415 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.314374    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" podStartSLOduration=1.3143613699999999 podStartE2EDuration="1.31436137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:12.312283853 +0000 UTC m=+1016.483358465" watchObservedRunningTime="2026-09-04 23:02:12.31436137 +0000 UTC m=+1016.485436021"
Sep 04 23:02:12.325813 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.325790    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d629a3-edc0-410d-aa4a-44cb056bdfb5" path="/var/lib/kubelet/pods/31d629a3-edc0-410d-aa4a-44cb056bdfb5/volumes"
Sep 04 23:02:12.328250 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:12.328212    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" podStartSLOduration=1.328199205 podStartE2EDuration="1.328199205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:12.327484246 +0000 UTC m=+1016.498558859" watchObservedRunningTime="2026-09-04 23:02:12.328199205 +0000 UTC m=+1016.499273818"
Sep 04 23:02:15.260218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:15.260181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:18.700233 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:18.700194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:02:20.336904 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.336871    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d236955-e765-4539-a04e-7da81e9b36c1" containerID="151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e" exitCode=0
Sep 04 23:02:20.337306 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.336944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerDied","Data":"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"}
Sep 04 23:02:20.337306 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.337300    2577 scope.go:117] "RemoveContainer" containerID="151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"
Sep 04 23:02:20.338368 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.338336    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:02:20.338480 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.338407    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:02:20.338480 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:20.338441    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:02:21.346495 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:21.346453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerStarted","Data":"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"}
Sep 04 23:02:21.787354 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:21.787264    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:21.787526 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:21.787467    2577 patch_prober.go:28] interesting pod/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:02:21.787598 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:21.787532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:02:22.787644 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:22.787605    2577 patch_prober.go:28] interesting pod/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:02:22.788041 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:22.787676    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:02:23.305175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:23.305140    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:02:23.305359 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:23.305221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:02:23.787587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:23.787543    2577 patch_prober.go:28] interesting pod/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:02:23.787805 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:23.787612    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:02:24.787288 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:24.787247    2577 patch_prober.go:28] interesting pod/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:02:24.787464 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:24.787314    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:02:25.787395 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:25.787356    2577 patch_prober.go:28] interesting pod/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Sep 04 23:02:25.787822 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:25.787416    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Sep 04 23:02:27.048231 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:27.048195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:27.048879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:27.048625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:27.049447 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:27.049426    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:32.185103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.185068    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:32.185706 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.185287    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" podUID="cd4bdb92-8f36-4456-899b-abac40878869" containerName="payload-pre-processing" containerID="cri-o://a2f195e8559b9da271655e2586948cb8e1485661224b68cb5b54d3259faca747" gracePeriod=30
Sep 04 23:02:32.208903 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.208876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:02:32.405943 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.405910    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd4bdb92-8f36-4456-899b-abac40878869" containerID="a2f195e8559b9da271655e2586948cb8e1485661224b68cb5b54d3259faca747" exitCode=0
Sep 04 23:02:32.406116 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.405984    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" event={"ID":"cd4bdb92-8f36-4456-899b-abac40878869","Type":"ContainerDied","Data":"a2f195e8559b9da271655e2586948cb8e1485661224b68cb5b54d3259faca747"}
Sep 04 23:02:32.434880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.434859    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:32.603112 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.603083    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume\") pod \"cd4bdb92-8f36-4456-899b-abac40878869\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") "
Sep 04 23:02:32.603260 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.603187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bdsh\" (UniqueName: \"kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh\") pod \"cd4bdb92-8f36-4456-899b-abac40878869\" (UID: \"cd4bdb92-8f36-4456-899b-abac40878869\") "
Sep 04 23:02:32.603501 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.603481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd4bdb92-8f36-4456-899b-abac40878869" (UID: "cd4bdb92-8f36-4456-899b-abac40878869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:02:32.605185 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.605158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh" (OuterVolumeSpecName: "kube-api-access-6bdsh") pod "cd4bdb92-8f36-4456-899b-abac40878869" (UID: "cd4bdb92-8f36-4456-899b-abac40878869"). InnerVolumeSpecName "kube-api-access-6bdsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:32.704580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.704541    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bdsh\" (UniqueName: \"kubernetes.io/projected/cd4bdb92-8f36-4456-899b-abac40878869-kube-api-access-6bdsh\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:32.704580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:32.704572    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd4bdb92-8f36-4456-899b-abac40878869-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:33.414175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.414146    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"
Sep 04 23:02:33.414175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.414167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84" event={"ID":"cd4bdb92-8f36-4456-899b-abac40878869","Type":"ContainerDied","Data":"18db517128b31ec47edb72d46cc1cd8486058d2404b86bf81039c959269899c4"}
Sep 04 23:02:33.414663 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.414231    2577 scope.go:117] "RemoveContainer" containerID="a2f195e8559b9da271655e2586948cb8e1485661224b68cb5b54d3259faca747"
Sep 04 23:02:33.438195 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.438168    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:33.442433 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.442407    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e8480dce-b9f4dbcf4-gsd84"]
Sep 04 23:02:33.513150 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.513108    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:02:33.513404 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:33.513364    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy" containerID="cri-o://0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5" gracePeriod=30
Sep 04 23:02:34.325322 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:34.325292    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd4bdb92-8f36-4456-899b-abac40878869" path="/var/lib/kubelet/pods/cd4bdb92-8f36-4456-899b-abac40878869/volumes"
Sep 04 23:02:35.174909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.174871    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:02:35.175288 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.175179    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4bdb92-8f36-4456-899b-abac40878869" containerName="payload-pre-processing"
Sep 04 23:02:35.175288 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.175192    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4bdb92-8f36-4456-899b-abac40878869" containerName="payload-pre-processing"
Sep 04 23:02:35.175288 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.175258    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4bdb92-8f36-4456-899b-abac40878869" containerName="payload-pre-processing"
Sep 04 23:02:35.179442 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.179423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.181939 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.181909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fd6f6\"/\"openshift-service-ca.crt\""
Sep 04 23:02:35.182080 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.182054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fd6f6\"/\"kube-root-ca.crt\""
Sep 04 23:02:35.182933 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.182915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fd6f6\"/\"default-dockercfg-bxcvg\""
Sep 04 23:02:35.183023 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.182931    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fd6f6\"/\"test-model-4fd6f6-kserve-self-signed-certs\""
Sep 04 23:02:35.190299 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.190276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:02:35.331102 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.331102 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctxx\" (UniqueName: \"kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.331344 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.331344 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.331344 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.331344 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.331305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.431976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctxx\" (UniqueName: \"kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432404 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432404 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.432629 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.432581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.434320 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.434297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.434718 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.434700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.440076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.440054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctxx\" (UniqueName: \"kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx\") pod \"test-model-4fd6f6-kserve-bc7df5cdf-xl72d\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") " pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.488971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.488942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:35.757821 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:35.757796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:02:35.760072 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:35.760040    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29d6f8a3_afef_476e_9186_d6a61a2556b2.slice/crio-6452e694fdd79eaa45cdfcc083f443335d16c05af17e022613e00371c1f2a7de WatchSource:0}: Error finding container 6452e694fdd79eaa45cdfcc083f443335d16c05af17e022613e00371c1f2a7de: Status 404 returned error can't find the container with id 6452e694fdd79eaa45cdfcc083f443335d16c05af17e022613e00371c1f2a7de
Sep 04 23:02:36.429798 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:36.429762    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" event={"ID":"29d6f8a3-afef-476e-9186-d6a61a2556b2","Type":"ContainerStarted","Data":"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"}
Sep 04 23:02:36.429798 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:36.429805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" event={"ID":"29d6f8a3-afef-476e-9186-d6a61a2556b2","Type":"ContainerStarted","Data":"6452e694fdd79eaa45cdfcc083f443335d16c05af17e022613e00371c1f2a7de"}
Sep 04 23:02:36.447157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:36.447104    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" podStartSLOduration=1.447088519 podStartE2EDuration="1.447088519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:36.445858382 +0000 UTC m=+1040.616932993" watchObservedRunningTime="2026-09-04 23:02:36.447088519 +0000 UTC m=+1040.618163168"
Sep 04 23:02:38.758265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.758237    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:38.793628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.793545    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:02:38.794063 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794034    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.794106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.794106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794088    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.794106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.794203 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794193    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.794246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.794209    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" containerName="istio-proxy"
Sep 04 23:02:38.797296 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.797275    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.799804 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.799784    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8e65e509\""
Sep 04 23:02:38.808454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.808432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:02:38.842943 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.842914    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:02:38.846072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.846057    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.848416 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.848395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8e65e509-dockercfg-lrkkl\""
Sep 04 23:02:38.856248 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.856222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:02:38.860819 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.860921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.860921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860853    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.860921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861039 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860923    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861039 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s9b5\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861039 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.860992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861184 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861184 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:38.861184 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs\") pod \"2d236955-e765-4539-a04e-7da81e9b36c1\" (UID: \"2d236955-e765-4539-a04e-7da81e9b36c1\") "
Sep 04 23:02:38.861286 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861183    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:38.861327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.861387 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.861439 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnsp\" (UniqueName: \"kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.861508 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kz9z\" (UniqueName: \"kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.861584 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:38.861584 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861568    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-credential-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.861698 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:38.861816 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861798    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.861978 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.861954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:02:38.863284 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.863259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:38.863381 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.863282    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5" (OuterVolumeSpecName: "kube-api-access-9s9b5") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "kube-api-access-9s9b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:02:38.863540 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.863519    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:02:38.863540 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.863528    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d236955-e765-4539-a04e-7da81e9b36c1" (UID: "2d236955-e765-4539-a04e-7da81e9b36c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:02:38.962383 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.962582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.962582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnsp\" (UniqueName: \"kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.962582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kz9z\" (UniqueName: \"kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.962582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962556    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d236955-e765-4539-a04e-7da81e9b36c1-istio-podinfo\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962572    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962585    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-envoy\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962598    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9b5\" (UniqueName: \"kubernetes.io/projected/2d236955-e765-4539-a04e-7da81e9b36c1-kube-api-access-9s9b5\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962610    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-istio-data\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962626    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d236955-e765-4539-a04e-7da81e9b36c1-istiod-ca-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.962866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.962638    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d236955-e765-4539-a04e-7da81e9b36c1-workload-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:02:38.963317 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.963294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.963354 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.963294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:38.970395 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.970369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnsp\" (UniqueName: \"kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp\") pod \"payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:38.970508 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:38.970370    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kz9z\" (UniqueName: \"kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z\") pod \"payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") " pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:39.107598 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.107558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:39.156108 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.156073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:39.376523 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.376501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:02:39.378204 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:39.378174    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1d45c3_e78d_4fcd_930e_aa1e5aed1977.slice/crio-ec8c9ed43d07c1993fad572dfcfde55894f0c2ce7927be69cc1bd932b8284aca WatchSource:0}: Error finding container ec8c9ed43d07c1993fad572dfcfde55894f0c2ce7927be69cc1bd932b8284aca: Status 404 returned error can't find the container with id ec8c9ed43d07c1993fad572dfcfde55894f0c2ce7927be69cc1bd932b8284aca
Sep 04 23:02:39.441538 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.441513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:02:39.446338 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:39.446037    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode099d799_b328_43f2_9b41_71b9b4a9192d.slice/crio-9dcc012ba71cd803ce9f1466968ad636a11435044792930f1be467ed9833c78b WatchSource:0}: Error finding container 9dcc012ba71cd803ce9f1466968ad636a11435044792930f1be467ed9833c78b: Status 404 returned error can't find the container with id 9dcc012ba71cd803ce9f1466968ad636a11435044792930f1be467ed9833c78b
Sep 04 23:02:39.449795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.449767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" event={"ID":"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977","Type":"ContainerStarted","Data":"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"}
Sep 04 23:02:39.449910 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.449821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" event={"ID":"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977","Type":"ContainerStarted","Data":"ec8c9ed43d07c1993fad572dfcfde55894f0c2ce7927be69cc1bd932b8284aca"}
Sep 04 23:02:39.451470 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.451447    2577 generic.go:356] "Generic (PLEG): container finished" podID="2d236955-e765-4539-a04e-7da81e9b36c1" containerID="0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5" exitCode=0
Sep 04 23:02:39.451587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.451480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerDied","Data":"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"}
Sep 04 23:02:39.451587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.451489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"
Sep 04 23:02:39.451587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.451511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm" event={"ID":"2d236955-e765-4539-a04e-7da81e9b36c1","Type":"ContainerDied","Data":"16e85029f83dd044a1833e323d278906dec48a1b6f9e07565fa1589abee69a2c"}
Sep 04 23:02:39.451587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.451531    2577 scope.go:117] "RemoveContainer" containerID="0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"
Sep 04 23:02:39.452444 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.452424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:39.462112 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.462032    2577 scope.go:117] "RemoveContainer" containerID="151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"
Sep 04 23:02:39.471994 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.471956    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" podStartSLOduration=1.471943209 podStartE2EDuration="1.471943209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:39.470196912 +0000 UTC m=+1043.641271523" watchObservedRunningTime="2026-09-04 23:02:39.471943209 +0000 UTC m=+1043.643017821"
Sep 04 23:02:39.472585 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.472569    2577 scope.go:117] "RemoveContainer" containerID="0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"
Sep 04 23:02:39.472896 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:39.472873    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5\": container with ID starting with 0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5 not found: ID does not exist" containerID="0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"
Sep 04 23:02:39.472984 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.472902    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5"} err="failed to get container status \"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5\": rpc error: code = NotFound desc = could not find container \"0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5\": container with ID starting with 0753b1fa0a358b95cc2e3b400aeab5030e838239a35f9af2700c4c0eaab2bef5 not found: ID does not exist"
Sep 04 23:02:39.472984 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.472919    2577 scope.go:117] "RemoveContainer" containerID="151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"
Sep 04 23:02:39.473178 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:02:39.473163    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e\": container with ID starting with 151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e not found: ID does not exist" containerID="151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"
Sep 04 23:02:39.473221 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.473183    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e"} err="failed to get container status \"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e\": rpc error: code = NotFound desc = could not find container \"151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e\": container with ID starting with 151a870743e5bd2286e0a2f6606ba87314c6d3565c20e11e96b0fbfabf590c7e not found: ID does not exist"
Sep 04 23:02:39.484593 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.484573    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:02:39.488672 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:39.488653    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e8480dce-openshift-default-5df6869cc4-5lpnm"]
Sep 04 23:02:40.342017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:40.341974    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d236955-e765-4539-a04e-7da81e9b36c1" path="/var/lib/kubelet/pods/2d236955-e765-4539-a04e-7da81e9b36c1/volumes"
Sep 04 23:02:40.459792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:40.459753    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" event={"ID":"e099d799-b328-43f2-9b41-71b9b4a9192d","Type":"ContainerStarted","Data":"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"}
Sep 04 23:02:40.459985 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:40.459816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" event={"ID":"e099d799-b328-43f2-9b41-71b9b4a9192d","Type":"ContainerStarted","Data":"9dcc012ba71cd803ce9f1466968ad636a11435044792930f1be467ed9833c78b"}
Sep 04 23:02:40.460076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:40.460059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:40.481018 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:40.480936    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" podStartSLOduration=2.480916881 podStartE2EDuration="2.480916881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:40.478015307 +0000 UTC m=+1044.649089920" watchObservedRunningTime="2026-09-04 23:02:40.480916881 +0000 UTC m=+1044.651991494"
Sep 04 23:02:45.489577 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:45.489539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:45.490144 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:45.489599    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:45.501877 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:45.501847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:46.497530 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:46.497500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:02:49.191915 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:49.191876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:02:50.461983 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:50.461952    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:02:51.466944 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:51.466912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:02:55.389284 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.389242    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:02:55.392667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.392648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.395910 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.395884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-72563d\"/\"explicit-model-72563d-kserve-self-signed-certs\""
Sep 04 23:02:55.396026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.395886    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-72563d\"/\"kube-root-ca.crt\""
Sep 04 23:02:55.396026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.395951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-72563d\"/\"openshift-service-ca.crt\""
Sep 04 23:02:55.396026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.396011    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-72563d\"/\"default-dockercfg-l7dq4\""
Sep 04 23:02:55.402855 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.402835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:02:55.511514 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.511692 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.511692 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.511692 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.511692 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9782\" (UniqueName: \"kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.511884 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.511697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613113 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613113 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9782\" (UniqueName: \"kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613579 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613579 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.613651 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.613629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.615436 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.615419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.615695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.615679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.621531 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.621506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9782\" (UniqueName: \"kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782\") pod \"explicit-model-72563d-kserve-59cc76cc79-vr8ht\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") " pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.703971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.703861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:02:55.974887 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:55.974858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:02:55.976879 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:02:55.976845    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9b2b776_bd9e_4d52_bf37_ea96a082d726.slice/crio-796c706bf690f2997e816ec289bc0cb67493d528102e1ba1adea57caf4e4d703 WatchSource:0}: Error finding container 796c706bf690f2997e816ec289bc0cb67493d528102e1ba1adea57caf4e4d703: Status 404 returned error can't find the container with id 796c706bf690f2997e816ec289bc0cb67493d528102e1ba1adea57caf4e4d703
Sep 04 23:02:56.532468 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:56.532430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" event={"ID":"a9b2b776-bd9e-4d52-bf37-ea96a082d726","Type":"ContainerStarted","Data":"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"}
Sep 04 23:02:56.532862 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:56.532476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" event={"ID":"a9b2b776-bd9e-4d52-bf37-ea96a082d726","Type":"ContainerStarted","Data":"796c706bf690f2997e816ec289bc0cb67493d528102e1ba1adea57caf4e4d703"}
Sep 04 23:02:56.550554 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:02:56.550504    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" podStartSLOduration=1.550488513 podStartE2EDuration="1.550488513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:02:56.549392707 +0000 UTC m=+1060.720467346" watchObservedRunningTime="2026-09-04 23:02:56.550488513 +0000 UTC m=+1060.721563125"
Sep 04 23:03:05.704758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:05.704706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:05.705259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:05.704787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:05.717289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:05.717263    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:06.589387 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:06.589350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:06.640293 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:06.640258    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:03:06.726714 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:06.726680    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:06.727127 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:06.726789    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:07.226772978 +0000 UTC m=+1071.397847568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:07.231288 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:07.231248    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:07.231474 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:07.231334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:08.231319533 +0000 UTC m=+1072.402394122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:08.242097 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:08.241992    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:08.242097 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:08.242081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:10.242067018 +0000 UTC m=+1074.413141609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:09.025863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.025828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:03:09.026081 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.026046    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" podUID="e099d799-b328-43f2-9b41-71b9b4a9192d" containerName="payload-processing" containerID="cri-o://8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43" gracePeriod=30
Sep 04 23:03:09.032831 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.032796    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:03:09.033564 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.033529    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" podUID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" containerName="payload-pre-processing" containerID="cri-o://4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7" gracePeriod=30
Sep 04 23:03:09.300368 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.300340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:03:09.303679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.303658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:03:09.452694 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.452664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kz9z\" (UniqueName: \"kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z\") pod \"e099d799-b328-43f2-9b41-71b9b4a9192d\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") "
Sep 04 23:03:09.452913 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.452702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume\") pod \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") "
Sep 04 23:03:09.452913 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.452758    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnsp\" (UniqueName: \"kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp\") pod \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\" (UID: \"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977\") "
Sep 04 23:03:09.452913 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.452781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume\") pod \"e099d799-b328-43f2-9b41-71b9b4a9192d\" (UID: \"e099d799-b328-43f2-9b41-71b9b4a9192d\") "
Sep 04 23:03:09.453235 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.453205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e099d799-b328-43f2-9b41-71b9b4a9192d" (UID: "e099d799-b328-43f2-9b41-71b9b4a9192d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:03:09.453362 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.453203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" (UID: "0b1d45c3-e78d-4fcd-930e-aa1e5aed1977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:03:09.454897 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.454873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z" (OuterVolumeSpecName: "kube-api-access-9kz9z") pod "e099d799-b328-43f2-9b41-71b9b4a9192d" (UID: "e099d799-b328-43f2-9b41-71b9b4a9192d"). InnerVolumeSpecName "kube-api-access-9kz9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:09.454897 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.454885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp" (OuterVolumeSpecName: "kube-api-access-jhnsp") pod "0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" (UID: "0b1d45c3-e78d-4fcd-930e-aa1e5aed1977"). InnerVolumeSpecName "kube-api-access-jhnsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:09.553535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.553441    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnsp\" (UniqueName: \"kubernetes.io/projected/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-kube-api-access-jhnsp\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:09.553535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.553477    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e099d799-b328-43f2-9b41-71b9b4a9192d-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:09.553535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.553487    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kz9z\" (UniqueName: \"kubernetes.io/projected/e099d799-b328-43f2-9b41-71b9b4a9192d-kube-api-access-9kz9z\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:09.553535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.553495    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:09.595880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.595845    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" containerID="4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7" exitCode=0
Sep 04 23:03:09.596029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.595891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"
Sep 04 23:03:09.596029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.595921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" event={"ID":"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977","Type":"ContainerDied","Data":"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"}
Sep 04 23:03:09.596029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.595961    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v" event={"ID":"0b1d45c3-e78d-4fcd-930e-aa1e5aed1977","Type":"ContainerDied","Data":"ec8c9ed43d07c1993fad572dfcfde55894f0c2ce7927be69cc1bd932b8284aca"}
Sep 04 23:03:09.596029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.595983    2577 scope.go:117] "RemoveContainer" containerID="4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"
Sep 04 23:03:09.597311 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.597245    2577 generic.go:356] "Generic (PLEG): container finished" podID="e099d799-b328-43f2-9b41-71b9b4a9192d" containerID="8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43" exitCode=0
Sep 04 23:03:09.597311 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.597286    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"
Sep 04 23:03:09.597479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.597340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" event={"ID":"e099d799-b328-43f2-9b41-71b9b4a9192d","Type":"ContainerDied","Data":"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"}
Sep 04 23:03:09.597479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.597390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp" event={"ID":"e099d799-b328-43f2-9b41-71b9b4a9192d","Type":"ContainerDied","Data":"9dcc012ba71cd803ce9f1466968ad636a11435044792930f1be467ed9833c78b"}
Sep 04 23:03:09.605953 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.605937    2577 scope.go:117] "RemoveContainer" containerID="4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"
Sep 04 23:03:09.606407 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:09.606297    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7\": container with ID starting with 4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7 not found: ID does not exist" containerID="4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"
Sep 04 23:03:09.606407 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.606334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7"} err="failed to get container status \"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7\": rpc error: code = NotFound desc = could not find container \"4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7\": container with ID starting with 4c56174984c660bd1b4e8e2cb20203e47e5f62b6050d1cb9a41ff82e42492ce7 not found: ID does not exist"
Sep 04 23:03:09.606407 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.606357    2577 scope.go:117] "RemoveContainer" containerID="8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"
Sep 04 23:03:09.614383 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.614368    2577 scope.go:117] "RemoveContainer" containerID="8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"
Sep 04 23:03:09.614617 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:09.614599    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43\": container with ID starting with 8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43 not found: ID does not exist" containerID="8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"
Sep 04 23:03:09.614660 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.614623    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43"} err="failed to get container status \"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43\": rpc error: code = NotFound desc = could not find container \"8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43\": container with ID starting with 8e3dde56184e06155a0866fa5a92b6898ee74fe89fb86a9e3fd8f75eddaa0d43 not found: ID does not exist"
Sep 04 23:03:09.617320 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.617297    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:03:09.621247 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.621228    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8e65e509-6ff98d9ddb-6cr8v"]
Sep 04 23:03:09.631897 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.631876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:03:09.635560 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:09.635542    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8e65e509-69ddf57949-hkkmp"]
Sep 04 23:03:10.261206 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:10.261167    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:10.261393 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:10.261247    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:14.261233233 +0000 UTC m=+1078.432307823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:10.324962 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:10.324929    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" path="/var/lib/kubelet/pods/0b1d45c3-e78d-4fcd-930e-aa1e5aed1977/volumes"
Sep 04 23:03:10.325333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:10.325289    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e099d799-b328-43f2-9b41-71b9b4a9192d" path="/var/lib/kubelet/pods/e099d799-b328-43f2-9b41-71b9b4a9192d/volumes"
Sep 04 23:03:12.896726 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:12.896692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:03:14.293392 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:14.293358    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:14.293814 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:14.293427    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:22.293412295 +0000 UTC m=+1086.464486885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:17.649581 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.649545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:03:17.650078 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.649838    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" podUID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" containerName="payload-processing" containerID="cri-o://50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9" gracePeriod=30
Sep 04 23:03:17.657446 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.657417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:03:17.657672 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.657646    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" podUID="29d457d9-728f-47e3-90df-ddbc9efe16d7" containerName="payload-pre-processing" containerID="cri-o://3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1" gracePeriod=30
Sep 04 23:03:17.923149 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.923124    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:03:17.925793 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:17.925770    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:03:18.027279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume\") pod \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") "
Sep 04 23:03:18.027457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gxhm\" (UniqueName: \"kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm\") pod \"29d457d9-728f-47e3-90df-ddbc9efe16d7\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") "
Sep 04 23:03:18.027457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfvrl\" (UniqueName: \"kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl\") pod \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\" (UID: \"5d6db88d-f65d-4ad2-a218-0925b6aaef06\") "
Sep 04 23:03:18.027457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume\") pod \"29d457d9-728f-47e3-90df-ddbc9efe16d7\" (UID: \"29d457d9-728f-47e3-90df-ddbc9efe16d7\") "
Sep 04 23:03:18.027683 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027652    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d6db88d-f65d-4ad2-a218-0925b6aaef06" (UID: "5d6db88d-f65d-4ad2-a218-0925b6aaef06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:03:18.027872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.027803    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29d457d9-728f-47e3-90df-ddbc9efe16d7" (UID: "29d457d9-728f-47e3-90df-ddbc9efe16d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:03:18.029446 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.029425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl" (OuterVolumeSpecName: "kube-api-access-rfvrl") pod "5d6db88d-f65d-4ad2-a218-0925b6aaef06" (UID: "5d6db88d-f65d-4ad2-a218-0925b6aaef06"). InnerVolumeSpecName "kube-api-access-rfvrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:18.029536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.029468    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm" (OuterVolumeSpecName: "kube-api-access-9gxhm") pod "29d457d9-728f-47e3-90df-ddbc9efe16d7" (UID: "29d457d9-728f-47e3-90df-ddbc9efe16d7"). InnerVolumeSpecName "kube-api-access-9gxhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:18.128617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.128582    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6db88d-f65d-4ad2-a218-0925b6aaef06-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:18.128617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.128611    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gxhm\" (UniqueName: \"kubernetes.io/projected/29d457d9-728f-47e3-90df-ddbc9efe16d7-kube-api-access-9gxhm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:18.128617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.128624    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvrl\" (UniqueName: \"kubernetes.io/projected/5d6db88d-f65d-4ad2-a218-0925b6aaef06-kube-api-access-rfvrl\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:18.128887 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.128636    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d457d9-728f-47e3-90df-ddbc9efe16d7-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:18.640502 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.640412    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" containerID="50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9" exitCode=0
Sep 04 23:03:18.640502 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.640470    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"
Sep 04 23:03:18.640708 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.640505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" event={"ID":"5d6db88d-f65d-4ad2-a218-0925b6aaef06","Type":"ContainerDied","Data":"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"}
Sep 04 23:03:18.640708 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.640591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2" event={"ID":"5d6db88d-f65d-4ad2-a218-0925b6aaef06","Type":"ContainerDied","Data":"e3d2411fec1b9ea41665965e8efafc20b1e2b122fb590eecb07871bc589cef4d"}
Sep 04 23:03:18.640708 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.640618    2577 scope.go:117] "RemoveContainer" containerID="50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"
Sep 04 23:03:18.641903 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.641880    2577 generic.go:356] "Generic (PLEG): container finished" podID="29d457d9-728f-47e3-90df-ddbc9efe16d7" containerID="3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1" exitCode=0
Sep 04 23:03:18.642034 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.641911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" event={"ID":"29d457d9-728f-47e3-90df-ddbc9efe16d7","Type":"ContainerDied","Data":"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"}
Sep 04 23:03:18.642034 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.641953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"
Sep 04 23:03:18.642034 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.641965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb" event={"ID":"29d457d9-728f-47e3-90df-ddbc9efe16d7","Type":"ContainerDied","Data":"d163e250de101101868669f4882203cca14edc2cff6423cf3dc7ab8d9d4befb2"}
Sep 04 23:03:18.649131 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.649114    2577 scope.go:117] "RemoveContainer" containerID="50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"
Sep 04 23:03:18.649396 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:18.649374    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9\": container with ID starting with 50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9 not found: ID does not exist" containerID="50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"
Sep 04 23:03:18.649464 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.649399    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9"} err="failed to get container status \"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9\": rpc error: code = NotFound desc = could not find container \"50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9\": container with ID starting with 50d2a4323f21936d4e58b76f7525597cd072eac718e72ceaae877734d48bf6c9 not found: ID does not exist"
Sep 04 23:03:18.649464 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.649415    2577 scope.go:117] "RemoveContainer" containerID="3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"
Sep 04 23:03:18.656597 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.656578    2577 scope.go:117] "RemoveContainer" containerID="3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"
Sep 04 23:03:18.656865 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:18.656847    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1\": container with ID starting with 3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1 not found: ID does not exist" containerID="3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"
Sep 04 23:03:18.656902 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.656868    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1"} err="failed to get container status \"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1\": rpc error: code = NotFound desc = could not find container \"3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1\": container with ID starting with 3c58ec81e97d756598df548f58eb97235398512b64157a620e73ebec15f459b1 not found: ID does not exist"
Sep 04 23:03:18.662504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.662486    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:03:18.667422 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.667403    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-72563d-85bb7795d-4vlxb"]
Sep 04 23:03:18.684833 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.684809    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:03:18.688599 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:18.688579    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-72563d-556c84c889-vx8b2"]
Sep 04 23:03:20.324505 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:20.324472    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29d457d9-728f-47e3-90df-ddbc9efe16d7" path="/var/lib/kubelet/pods/29d457d9-728f-47e3-90df-ddbc9efe16d7/volumes"
Sep 04 23:03:20.324908 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:20.324837    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" path="/var/lib/kubelet/pods/5d6db88d-f65d-4ad2-a218-0925b6aaef06/volumes"
Sep 04 23:03:22.365678 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:22.365636    2577 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-self-signed-certs: secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:22.366092 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:22.365720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs podName:a9b2b776-bd9e-4d52-bf37-ea96a082d726 nodeName:}" failed. No retries permitted until 2026-09-04 23:03:38.365705981 +0000 UTC m=+1102.536780570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs") pod "explicit-model-72563d-kserve-59cc76cc79-vr8ht" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726") : secret "explicit-model-72563d-kserve-self-signed-certs" not found
Sep 04 23:03:23.608792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.608755    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" podUID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" containerName="main" containerID="cri-o://f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b" gracePeriod=45
Sep 04 23:03:23.855027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.855003    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:23.980098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9782\" (UniqueName: \"kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980439 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980439 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980197    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir\") pod \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\" (UID: \"a9b2b776-bd9e-4d52-bf37-ea96a082d726\") "
Sep 04 23:03:23.980439 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:23.980610 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980464    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:23.980610 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:23.980705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.980612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home" (OuterVolumeSpecName: "home") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:23.982346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.982323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm" (OuterVolumeSpecName: "dshm") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:23.982423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.982402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:03:23.982459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:23.982419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782" (OuterVolumeSpecName: "kube-api-access-m9782") pod "a9b2b776-bd9e-4d52-bf37-ea96a082d726" (UID: "a9b2b776-bd9e-4d52-bf37-ea96a082d726"). InnerVolumeSpecName "kube-api-access-m9782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:24.081068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.081027    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:24.081068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.081059    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9782\" (UniqueName: \"kubernetes.io/projected/a9b2b776-bd9e-4d52-bf37-ea96a082d726-kube-api-access-m9782\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:24.081068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.081069    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:24.081068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.081077    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:24.081333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.081086    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9b2b776-bd9e-4d52-bf37-ea96a082d726-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:24.670951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.670857    2577 generic.go:356] "Generic (PLEG): container finished" podID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" containerID="f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b" exitCode=0
Sep 04 23:03:24.670951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.670904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" event={"ID":"a9b2b776-bd9e-4d52-bf37-ea96a082d726","Type":"ContainerDied","Data":"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"}
Sep 04 23:03:24.670951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.670924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"
Sep 04 23:03:24.670951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.670943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht" event={"ID":"a9b2b776-bd9e-4d52-bf37-ea96a082d726","Type":"ContainerDied","Data":"796c706bf690f2997e816ec289bc0cb67493d528102e1ba1adea57caf4e4d703"}
Sep 04 23:03:24.671532 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.670966    2577 scope.go:117] "RemoveContainer" containerID="f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"
Sep 04 23:03:24.679044 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.679022    2577 scope.go:117] "RemoveContainer" containerID="f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"
Sep 04 23:03:24.679291 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:24.679272    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b\": container with ID starting with f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b not found: ID does not exist" containerID="f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"
Sep 04 23:03:24.679358 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.679305    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b"} err="failed to get container status \"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b\": rpc error: code = NotFound desc = could not find container \"f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b\": container with ID starting with f51fd7633ba43c7bffc28d3890b5d7b83b6c531fd654ed585ec1e0c6f195a06b not found: ID does not exist"
Sep 04 23:03:24.687288 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.687266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:03:24.689403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:24.689383    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-72563d/explicit-model-72563d-kserve-59cc76cc79-vr8ht"]
Sep 04 23:03:26.325986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:26.325947    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" path="/var/lib/kubelet/pods/a9b2b776-bd9e-4d52-bf37-ea96a082d726/volumes"
Sep 04 23:03:29.194932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:29.194892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:03:30.010631 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.010596    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:03:30.011087 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011071    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e099d799-b328-43f2-9b41-71b9b4a9192d" containerName="payload-processing"
Sep 04 23:03:30.011131 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011091    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e099d799-b328-43f2-9b41-71b9b4a9192d" containerName="payload-processing"
Sep 04 23:03:30.011131 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011112    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" containerName="payload-pre-processing"
Sep 04 23:03:30.011131 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011122    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" containerName="payload-pre-processing"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d457d9-728f-47e3-90df-ddbc9efe16d7" containerName="payload-pre-processing"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011148    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d457d9-728f-47e3-90df-ddbc9efe16d7" containerName="payload-pre-processing"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011161    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" containerName="payload-processing"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011170    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" containerName="payload-processing"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011187    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" containerName="main"
Sep 04 23:03:30.011225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" containerName="main"
Sep 04 23:03:30.011403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011265    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1d45c3-e78d-4fcd-930e-aa1e5aed1977" containerName="payload-pre-processing"
Sep 04 23:03:30.011403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011276    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e099d799-b328-43f2-9b41-71b9b4a9192d" containerName="payload-processing"
Sep 04 23:03:30.011403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011287    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6db88d-f65d-4ad2-a218-0925b6aaef06" containerName="payload-processing"
Sep 04 23:03:30.011403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011297    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d457d9-728f-47e3-90df-ddbc9efe16d7" containerName="payload-pre-processing"
Sep 04 23:03:30.011403 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.011307    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b2b776-bd9e-4d52-bf37-ea96a082d726" containerName="main"
Sep 04 23:03:30.015744 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.015713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.018132 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.018109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-c72d2b-openshift-default-dockercfg-dnspq\""
Sep 04 23:03:30.026690 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.026663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:03:30.134466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.134834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.134753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56mv2\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235419 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56mv2\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.235861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.236241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.236241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.235941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.236241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.236131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.236352 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.236292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.238351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.238319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.238486 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.238469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.243092 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.243069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.243365 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.243344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56mv2\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2\") pod \"e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") " pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.326023 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.325939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:30.615988 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.615963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:03:30.617535 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:03:30.617508    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f201b2_6e83_4e8a_b4d0_10f99fa7c22e.slice/crio-82c93e8610b281e9808163a6d49352f134c7a324e3a81f95d58d69d75b2da96e WatchSource:0}: Error finding container 82c93e8610b281e9808163a6d49352f134c7a324e3a81f95d58d69d75b2da96e: Status 404 returned error can't find the container with id 82c93e8610b281e9808163a6d49352f134c7a324e3a81f95d58d69d75b2da96e
Sep 04 23:03:30.619849 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.619812    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:03:30.619932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.619885    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:03:30.619932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.619910    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:03:30.699931 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.699904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" event={"ID":"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e","Type":"ContainerStarted","Data":"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"}
Sep 04 23:03:30.700097 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.699964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" event={"ID":"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e","Type":"ContainerStarted","Data":"82c93e8610b281e9808163a6d49352f134c7a324e3a81f95d58d69d75b2da96e"}
Sep 04 23:03:30.727032 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:30.726984    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" podStartSLOduration=1.726967538 podStartE2EDuration="1.726967538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:03:30.725666084 +0000 UTC m=+1094.896740757" watchObservedRunningTime="2026-09-04 23:03:30.726967538 +0000 UTC m=+1094.898042150"
Sep 04 23:03:31.326148 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:31.326112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:31.327590 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:31.327562    2577 patch_prober.go:28] interesting pod/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 04 23:03:31.327764 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:31.327612    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 04 23:03:32.330460 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:32.330429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:32.708909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:32.708876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:32.709972 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:32.709953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:39.190566 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.190531    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:03:39.194023 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.194007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.196512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.196484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c72d2b\"/\"noten-model-c72d2b-kserve-self-signed-certs\""
Sep 04 23:03:39.196512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.196511    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c72d2b\"/\"openshift-service-ca.crt\""
Sep 04 23:03:39.197407 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.197385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c72d2b\"/\"default-dockercfg-q29d4\""
Sep 04 23:03:39.197515 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.197426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c72d2b\"/\"kube-root-ca.crt\""
Sep 04 23:03:39.203311 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.203287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:03:39.315225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.315225 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.315453 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.315453 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.315453 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2th\" (UniqueName: \"kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.315453 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.315382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2th\" (UniqueName: \"kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416559 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416614 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.416657 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.416631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.418476 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.418448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.418728 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.418710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.438213 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.438188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2th\" (UniqueName: \"kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th\") pod \"noten-model-c72d2b-kserve-f86d7974c-9cdck\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") " pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.504283 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.504180    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:39.776420 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:39.776393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:03:39.778571 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:03:39.778548    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8766d426_562e_4a77_8b63_9073f27649d3.slice/crio-d7119f2fb0b832af6f1f1de442e88dab0fbbc32a2037950adda51352e1f0f3c9 WatchSource:0}: Error finding container d7119f2fb0b832af6f1f1de442e88dab0fbbc32a2037950adda51352e1f0f3c9: Status 404 returned error can't find the container with id d7119f2fb0b832af6f1f1de442e88dab0fbbc32a2037950adda51352e1f0f3c9
Sep 04 23:03:40.746062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:40.746016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" event={"ID":"8766d426-562e-4a77-8b63-9073f27649d3","Type":"ContainerStarted","Data":"5540134e58166bdd5e012b203b70079a7dbe13e3072be6a776f0022592d6d1ad"}
Sep 04 23:03:40.746062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:40.746068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" event={"ID":"8766d426-562e-4a77-8b63-9073f27649d3","Type":"ContainerStarted","Data":"d7119f2fb0b832af6f1f1de442e88dab0fbbc32a2037950adda51352e1f0f3c9"}
Sep 04 23:03:40.765452 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:40.765406    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" podStartSLOduration=1.765391114 podStartE2EDuration="1.765391114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:03:40.76478982 +0000 UTC m=+1104.935864431" watchObservedRunningTime="2026-09-04 23:03:40.765391114 +0000 UTC m=+1104.936465722"
Sep 04 23:03:49.504596 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:49.504559    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:49.505007 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:49.504614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:49.517125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:49.517100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:49.794624 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:49.794538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:03:52.625266 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:52.625236    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:03:52.760724 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:52.760693    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:03:52.760923 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:52.760902    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerName="istio-proxy" containerID="cri-o://a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd" gracePeriod=30
Sep 04 23:03:57.999860 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:57.999833    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:58.078342 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078342 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078342 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078588 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078588 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078588 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078588 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078588 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078881 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56mv2\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2\") pod \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\" (UID: \"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e\") "
Sep 04 23:03:58.078881 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:58.078881 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:58.078881 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:58.079079 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.078981    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-credential-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.079079 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.079001    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.079079 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.079017    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-data\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.079079 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.079052    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:03:58.079219 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.079134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:58.080673 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.080644    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:03:58.080812 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.080668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2" (OuterVolumeSpecName: "kube-api-access-56mv2") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "kube-api-access-56mv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:58.080979 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.080957    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:03:58.080979 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.080967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" (UID: "e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:03:58.180251 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180215    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-envoy\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.180251 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180245    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.180251 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180255    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istio-podinfo\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.180497 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180265    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-istiod-ca-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.180497 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180273    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-workload-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.180497 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.180281    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56mv2\" (UniqueName: \"kubernetes.io/projected/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e-kube-api-access-56mv2\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:03:58.823891 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.823798    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerID="a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd" exitCode=0
Sep 04 23:03:58.823891 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.823856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"
Sep 04 23:03:58.824067 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.823892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" event={"ID":"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e","Type":"ContainerDied","Data":"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"}
Sep 04 23:03:58.824067 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.823967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf" event={"ID":"e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e","Type":"ContainerDied","Data":"82c93e8610b281e9808163a6d49352f134c7a324e3a81f95d58d69d75b2da96e"}
Sep 04 23:03:58.824067 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.823990    2577 scope.go:117] "RemoveContainer" containerID="a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"
Sep 04 23:03:58.832515 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.832501    2577 scope.go:117] "RemoveContainer" containerID="a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"
Sep 04 23:03:58.832809 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:03:58.832789    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd\": container with ID starting with a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd not found: ID does not exist" containerID="a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"
Sep 04 23:03:58.832869 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.832816    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd"} err="failed to get container status \"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd\": rpc error: code = NotFound desc = could not find container \"a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd\": container with ID starting with a76699c522f2f6345c2e701555b22a52f5bb21a373999ce804b64fc0956ce2dd not found: ID does not exist"
Sep 04 23:03:58.841633 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.841612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:03:58.848265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:03:58.848245    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-c72d2b-openshift-default-75b879dc64-4zgtf"]
Sep 04 23:04:00.196412 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.196383    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:04:00.325489 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.325458    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" path="/var/lib/kubelet/pods/e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e/volumes"
Sep 04 23:04:00.325719 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.325706    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"]
Sep 04 23:04:00.325991 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.325981    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerName="istio-proxy"
Sep 04 23:04:00.326030 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.325993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerName="istio-proxy"
Sep 04 23:04:00.326075 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.326061    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f201b2-6e83-4e8a-b4d0-10f99fa7c22e" containerName="istio-proxy"
Sep 04 23:04:00.329000 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.328970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:00.331629 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.331606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:04:00.331761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.331616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ncjwg\""
Sep 04 23:04:00.332282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.332260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"]
Sep 04 23:04:00.398519 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.398476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5\" (UID: \"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:00.499489 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.499394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5\" (UID: \"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:00.500014 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.499997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5\" (UID: \"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:00.641980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.641946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:00.907603 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:00.907571    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"]
Sep 04 23:04:00.910677 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:04:00.910650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f7b0f2_ee9a_49d7_9fc6_5e9158b65a75.slice/crio-9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab WatchSource:0}: Error finding container 9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab: Status 404 returned error can't find the container with id 9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab
Sep 04 23:04:01.838879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:01.838845    2577 generic.go:356] "Generic (PLEG): container finished" podID="d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" containerID="4df502874f4faf8b6fdc7e40e16487edb3829b86acf3d90b433d6769489b4b8d" exitCode=0
Sep 04 23:04:01.839255 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:01.838927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5" event={"ID":"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75","Type":"ContainerDied","Data":"4df502874f4faf8b6fdc7e40e16487edb3829b86acf3d90b433d6769489b4b8d"}
Sep 04 23:04:01.839255 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:01.838974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5" event={"ID":"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75","Type":"ContainerStarted","Data":"9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab"}
Sep 04 23:04:02.966351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:02.966325    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:03.021374 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.021346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca\") pod \"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75\" (UID: \"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75\") "
Sep 04 23:04:03.021705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.021682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" (UID: "d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:04:03.086872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.086835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:04:03.122925 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.122836    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75-maas-api-service-ca\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:03.848785 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.848717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5" event={"ID":"d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75","Type":"ContainerDied","Data":"9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab"}
Sep 04 23:04:03.848785 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.848772    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"
Sep 04 23:04:03.848958 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:03.848797    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9804b7ad7ad7a1564e35e113ecc7c32816cc17730df1849083e19ce3b0d885ab"
Sep 04 23:04:07.644423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.644381    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" podUID="8766d426-562e-4a77-8b63-9073f27649d3" containerName="main" containerID="cri-o://5540134e58166bdd5e012b203b70079a7dbe13e3072be6a776f0022592d6d1ad" gracePeriod=45
Sep 04 23:04:07.866969 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.866940    2577 generic.go:356] "Generic (PLEG): container finished" podID="8766d426-562e-4a77-8b63-9073f27649d3" containerID="5540134e58166bdd5e012b203b70079a7dbe13e3072be6a776f0022592d6d1ad" exitCode=0
Sep 04 23:04:07.867096 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.867013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" event={"ID":"8766d426-562e-4a77-8b63-9073f27649d3","Type":"ContainerDied","Data":"5540134e58166bdd5e012b203b70079a7dbe13e3072be6a776f0022592d6d1ad"}
Sep 04 23:04:07.881980 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.881957    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:04:07.965069 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.964975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965069 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965069 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965102    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj2th\" (UniqueName: \"kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th\") pod \"8766d426-562e-4a77-8b63-9073f27649d3\" (UID: \"8766d426-562e-4a77-8b63-9073f27649d3\") "
Sep 04 23:04:07.965475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:07.965475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home" (OuterVolumeSpecName: "home") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:07.965475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:07.965580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965531    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:07.965580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965550    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:07.965580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.965563    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:07.967330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.967308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:04:07.967408 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.967332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm" (OuterVolumeSpecName: "dshm") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:07.967408 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:07.967393    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th" (OuterVolumeSpecName: "kube-api-access-nj2th") pod "8766d426-562e-4a77-8b63-9073f27649d3" (UID: "8766d426-562e-4a77-8b63-9073f27649d3"). InnerVolumeSpecName "kube-api-access-nj2th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:04:08.065994 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.065960    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8766d426-562e-4a77-8b63-9073f27649d3-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:08.065994 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.065987    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8766d426-562e-4a77-8b63-9073f27649d3-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:08.065994 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.065996    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2th\" (UniqueName: \"kubernetes.io/projected/8766d426-562e-4a77-8b63-9073f27649d3-kube-api-access-nj2th\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:08.872542 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.872515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"
Sep 04 23:04:08.872956 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.872506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck" event={"ID":"8766d426-562e-4a77-8b63-9073f27649d3","Type":"ContainerDied","Data":"d7119f2fb0b832af6f1f1de442e88dab0fbbc32a2037950adda51352e1f0f3c9"}
Sep 04 23:04:08.872956 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.872637    2577 scope.go:117] "RemoveContainer" containerID="5540134e58166bdd5e012b203b70079a7dbe13e3072be6a776f0022592d6d1ad"
Sep 04 23:04:08.887690 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.887665    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:04:08.893510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:08.893486    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c72d2b/noten-model-c72d2b-kserve-f86d7974c-9cdck"]
Sep 04 23:04:10.325512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:10.325474    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8766d426-562e-4a77-8b63-9073f27649d3" path="/var/lib/kubelet/pods/8766d426-562e-4a77-8b63-9073f27649d3/volumes"
Sep 04 23:04:11.005160 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:11.005130    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"]
Sep 04 23:04:11.007333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:11.007311    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a64414-ng6w5"]
Sep 04 23:04:12.324783 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:12.324730    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" path="/var/lib/kubelet/pods/d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75/volumes"
Sep 04 23:04:13.593289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:13.593252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:04:15.216621 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.216570    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" podUID="29d6f8a3-afef-476e-9186-d6a61a2556b2" containerName="main" containerID="cri-o://d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68" gracePeriod=45
Sep 04 23:04:15.458466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.458440    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:04:15.528536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528457    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528503    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vctxx\" (UniqueName: \"kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528767 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528767 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528767 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528767 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs\") pod \"29d6f8a3-afef-476e-9186-d6a61a2556b2\" (UID: \"29d6f8a3-afef-476e-9186-d6a61a2556b2\") "
Sep 04 23:04:15.528985 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:15.528985 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528855    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:15.528985 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home" (OuterVolumeSpecName: "home") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:15.529125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.528986    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.529125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.529000    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.529125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.529012    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.530840 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.530819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm" (OuterVolumeSpecName: "dshm") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:15.530931 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.530895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:04:15.530976 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.530926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx" (OuterVolumeSpecName: "kube-api-access-vctxx") pod "29d6f8a3-afef-476e-9186-d6a61a2556b2" (UID: "29d6f8a3-afef-476e-9186-d6a61a2556b2"). InnerVolumeSpecName "kube-api-access-vctxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:04:15.629836 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.629807    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29d6f8a3-afef-476e-9186-d6a61a2556b2-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.629836 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.629833    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctxx\" (UniqueName: \"kubernetes.io/projected/29d6f8a3-afef-476e-9186-d6a61a2556b2-kube-api-access-vctxx\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.629836 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.629843    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29d6f8a3-afef-476e-9186-d6a61a2556b2-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:15.907895 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.907861    2577 generic.go:356] "Generic (PLEG): container finished" podID="29d6f8a3-afef-476e-9186-d6a61a2556b2" containerID="d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68" exitCode=0
Sep 04 23:04:15.908072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.907909    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"
Sep 04 23:04:15.908072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.907941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" event={"ID":"29d6f8a3-afef-476e-9186-d6a61a2556b2","Type":"ContainerDied","Data":"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"}
Sep 04 23:04:15.908072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.907989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d" event={"ID":"29d6f8a3-afef-476e-9186-d6a61a2556b2","Type":"ContainerDied","Data":"6452e694fdd79eaa45cdfcc083f443335d16c05af17e022613e00371c1f2a7de"}
Sep 04 23:04:15.908072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.908008    2577 scope.go:117] "RemoveContainer" containerID="d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"
Sep 04 23:04:15.916769 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.916731    2577 scope.go:117] "RemoveContainer" containerID="d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"
Sep 04 23:04:15.917050 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:04:15.917032    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68\": container with ID starting with d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68 not found: ID does not exist" containerID="d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"
Sep 04 23:04:15.917115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.917057    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68"} err="failed to get container status \"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68\": rpc error: code = NotFound desc = could not find container \"d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68\": container with ID starting with d610be09fd49e69b5d75460f45abdb4433826ada837d93bc4fb8a2bc385b8f68 not found: ID does not exist"
Sep 04 23:04:15.927747 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.927703    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:04:15.932024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:15.931998    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4fd6f6/test-model-4fd6f6-kserve-bc7df5cdf-xl72d"]
Sep 04 23:04:16.325535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:16.325460    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29d6f8a3-afef-476e-9186-d6a61a2556b2" path="/var/lib/kubelet/pods/29d6f8a3-afef-476e-9186-d6a61a2556b2/volumes"
Sep 04 23:04:28.718522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.718490    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:04:28.719020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.718772    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" podUID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" containerName="payload-pre-processing" containerID="cri-o://8a6b5c39e3fad9e7e96a4e3e22958185357a5eed43b6e0d703c9f593905d034f" gracePeriod=30
Sep 04 23:04:28.963557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.963526    2577 generic.go:356] "Generic (PLEG): container finished" podID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" containerID="8a6b5c39e3fad9e7e96a4e3e22958185357a5eed43b6e0d703c9f593905d034f" exitCode=0
Sep 04 23:04:28.963730 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.963609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" event={"ID":"31a04d3b-b317-4d43-8c58-c6d1633b4fd0","Type":"ContainerDied","Data":"8a6b5c39e3fad9e7e96a4e3e22958185357a5eed43b6e0d703c9f593905d034f"}
Sep 04 23:04:28.963730 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.963679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb" event={"ID":"31a04d3b-b317-4d43-8c58-c6d1633b4fd0","Type":"ContainerDied","Data":"1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3"}
Sep 04 23:04:28.963730 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.963705    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed26b293d362f473e2e497aa9b4308d5344d9f49bafbe2c81b0b4c14261ddd3"
Sep 04 23:04:28.975604 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:28.975556    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:04:29.045893 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.045860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume\") pod \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") "
Sep 04 23:04:29.046099 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.045932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sd6g\" (UniqueName: \"kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g\") pod \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\" (UID: \"31a04d3b-b317-4d43-8c58-c6d1633b4fd0\") "
Sep 04 23:04:29.046302 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.046276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31a04d3b-b317-4d43-8c58-c6d1633b4fd0" (UID: "31a04d3b-b317-4d43-8c58-c6d1633b4fd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:04:29.047993 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.047966    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g" (OuterVolumeSpecName: "kube-api-access-5sd6g") pod "31a04d3b-b317-4d43-8c58-c6d1633b4fd0" (UID: "31a04d3b-b317-4d43-8c58-c6d1633b4fd0"). InnerVolumeSpecName "kube-api-access-5sd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:04:29.147289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.147250    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd6g\" (UniqueName: \"kubernetes.io/projected/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-kube-api-access-5sd6g\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:29.147289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.147284    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a04d3b-b317-4d43-8c58-c6d1633b4fd0-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:29.967390 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.967358    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"
Sep 04 23:04:29.987224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.987200    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:04:29.991151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:29.991127    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fd6f6-55d8989bb-mwnfb"]
Sep 04 23:04:30.324856 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:30.324773    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" path="/var/lib/kubelet/pods/31a04d3b-b317-4d43-8c58-c6d1633b4fd0/volumes"
Sep 04 23:04:31.508401 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:31.506181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:04:35.265687 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:35.265656    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:04:35.266117 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:35.265924    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy" containerID="cri-o://9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce" gracePeriod=30
Sep 04 23:04:39.771593 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:39.771510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:04:40.514328 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.514304    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:04:40.648861 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.648823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.648890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bt6r\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.648915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.648955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.648984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649024 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token\") pod \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\" (UID: \"eb84129e-2316-47e5-8cbe-ab444e5f2f48\") "
Sep 04 23:04:40.649327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:40.649327 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:40.649592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:40.649592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649407    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-credential-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.649592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649426    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.649592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649439    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:04:40.649592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.649527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:40.651298 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.651269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:04:40.651421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.651302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:04:40.651475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.651464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r" (OuterVolumeSpecName: "kube-api-access-5bt6r") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "kube-api-access-5bt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:04:40.651551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.651533    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb84129e-2316-47e5-8cbe-ab444e5f2f48" (UID: "eb84129e-2316-47e5-8cbe-ab444e5f2f48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:04:40.749862 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749827    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-workload-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.749862 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749857    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-envoy\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.749862 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749866    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.750073 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749874    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-podinfo\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.750073 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749882    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bt6r\" (UniqueName: \"kubernetes.io/projected/eb84129e-2316-47e5-8cbe-ab444e5f2f48-kube-api-access-5bt6r\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.750073 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749891    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istiod-ca-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:40.750073 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:40.749899    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb84129e-2316-47e5-8cbe-ab444e5f2f48-istio-data\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:04:41.018282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.018185    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerID="9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce" exitCode=0
Sep 04 23:04:41.018282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.018232    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"
Sep 04 23:04:41.018282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.018267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerDied","Data":"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"}
Sep 04 23:04:41.018748 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.018320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9" event={"ID":"eb84129e-2316-47e5-8cbe-ab444e5f2f48","Type":"ContainerDied","Data":"4a03f4363c1710a7636359c1ee1653783b74c688278a7a38332af98c286a95cc"}
Sep 04 23:04:41.018748 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.018343    2577 scope.go:117] "RemoveContainer" containerID="9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"
Sep 04 23:04:41.027179 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.027164    2577 scope.go:117] "RemoveContainer" containerID="3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"
Sep 04 23:04:41.035022 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.034999    2577 scope.go:117] "RemoveContainer" containerID="9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"
Sep 04 23:04:41.035262 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:04:41.035244    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce\": container with ID starting with 9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce not found: ID does not exist" containerID="9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"
Sep 04 23:04:41.035333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.035275    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce"} err="failed to get container status \"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce\": rpc error: code = NotFound desc = could not find container \"9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce\": container with ID starting with 9d9e109f6f20a12d195a5df6ba673823b1d3cbc7d21f781710b99ce201faf4ce not found: ID does not exist"
Sep 04 23:04:41.035333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.035298    2577 scope.go:117] "RemoveContainer" containerID="3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"
Sep 04 23:04:41.035568 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:04:41.035546    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f\": container with ID starting with 3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f not found: ID does not exist" containerID="3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"
Sep 04 23:04:41.035652 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.035570    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f"} err="failed to get container status \"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f\": rpc error: code = NotFound desc = could not find container \"3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f\": container with ID starting with 3ee6d5165358d4ff936e6dbbbc139ebb6c26af01a0b34fe16b700b0464a4878f not found: ID does not exist"
Sep 04 23:04:41.044170 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.044150    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:04:41.050203 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.050185    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4fd6f6-openshift-default-756d8b8fc6-tq8b9"]
Sep 04 23:04:41.073500 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073476    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:04:41.073748 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" containerName="payload-pre-processing"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073750    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" containerName="payload-pre-processing"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073759    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29d6f8a3-afef-476e-9186-d6a61a2556b2" containerName="main"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073766    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="29d6f8a3-afef-476e-9186-d6a61a2556b2" containerName="main"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073774    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" containerName="revoke-keys"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073780    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" containerName="revoke-keys"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.073797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073793    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8766d426-562e-4a77-8b63-9073f27649d3" containerName="main"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8766d426-562e-4a77-8b63-9073f27649d3" containerName="main"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073815    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073820    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073866    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="29d6f8a3-afef-476e-9186-d6a61a2556b2" containerName="main"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073874    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073880    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a04d3b-b317-4d43-8c58-c6d1633b4fd0" containerName="payload-pre-processing"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073886    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" containerName="istio-proxy"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073892    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f7b0f2-ee9a-49d7-9fc6-5e9158b65a75" containerName="revoke-keys"
Sep 04 23:04:41.074008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.073898    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8766d426-562e-4a77-8b63-9073f27649d3" containerName="main"
Sep 04 23:04:41.078148 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.078132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.080377 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.080359    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-9ec1e5\""
Sep 04 23:04:41.087351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.087329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:04:41.104221 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.104196    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:04:41.107107 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.107093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.109691 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.109674    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-9ec1e5-dockercfg-n8r5r\""
Sep 04 23:04:41.117939 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.117920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:04:41.254387 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.254349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blsb\" (UniqueName: \"kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.254387 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.254388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vmq\" (UniqueName: \"kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.254592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.254430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.254592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.254495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.355392 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.355356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blsb\" (UniqueName: \"kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.355584 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.355398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vmq\" (UniqueName: \"kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.355584 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.355429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.355584 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.355472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.356175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.356156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.356247 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.356158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.363268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.363246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vmq\" (UniqueName: \"kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq\") pod \"payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.363434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.363281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blsb\" (UniqueName: \"kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb\") pod \"payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.388161 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.388136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:41.416629 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.416596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:41.658028 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.657951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:04:41.660163 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:04:41.660136    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ded5a1_c745_4b9a_92a6_7057d8c0f3bb.slice/crio-4518e7fecf88e149665cd59fcf2de18af94e7817a910c6344666ff7a563ded15 WatchSource:0}: Error finding container 4518e7fecf88e149665cd59fcf2de18af94e7817a910c6344666ff7a563ded15: Status 404 returned error can't find the container with id 4518e7fecf88e149665cd59fcf2de18af94e7817a910c6344666ff7a563ded15
Sep 04 23:04:41.694048 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:41.694019    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:04:41.696995 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:04:41.696970    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6acfd2e_b4d5_4300_8ea1_f0c21635d625.slice/crio-05f989852c8a4ee3da34380e3e81807b1b484eba46903bb01e6670b0429f8585 WatchSource:0}: Error finding container 05f989852c8a4ee3da34380e3e81807b1b484eba46903bb01e6670b0429f8585: Status 404 returned error can't find the container with id 05f989852c8a4ee3da34380e3e81807b1b484eba46903bb01e6670b0429f8585
Sep 04 23:04:42.028886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.028790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" event={"ID":"c6acfd2e-b4d5-4300-8ea1-f0c21635d625","Type":"ContainerStarted","Data":"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"}
Sep 04 23:04:42.028886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.028850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" event={"ID":"c6acfd2e-b4d5-4300-8ea1-f0c21635d625","Type":"ContainerStarted","Data":"05f989852c8a4ee3da34380e3e81807b1b484eba46903bb01e6670b0429f8585"}
Sep 04 23:04:42.029378 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.028996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:42.030646 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.030619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" event={"ID":"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb","Type":"ContainerStarted","Data":"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"}
Sep 04 23:04:42.030806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.030662    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" event={"ID":"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb","Type":"ContainerStarted","Data":"4518e7fecf88e149665cd59fcf2de18af94e7817a910c6344666ff7a563ded15"}
Sep 04 23:04:42.030806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.030683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:04:42.045615 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.045571    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" podStartSLOduration=1.0455570299999999 podStartE2EDuration="1.04555703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:04:42.044327131 +0000 UTC m=+1166.215401757" watchObservedRunningTime="2026-09-04 23:04:42.04555703 +0000 UTC m=+1166.216631636"
Sep 04 23:04:42.061373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.061319    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" podStartSLOduration=1.061299507 podStartE2EDuration="1.061299507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:04:42.059794324 +0000 UTC m=+1166.230868936" watchObservedRunningTime="2026-09-04 23:04:42.061299507 +0000 UTC m=+1166.232374123"
Sep 04 23:04:42.325797 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:42.325711    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb84129e-2316-47e5-8cbe-ab444e5f2f48" path="/var/lib/kubelet/pods/eb84129e-2316-47e5-8cbe-ab444e5f2f48/volumes"
Sep 04 23:04:51.160348 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.160314    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:04:51.163567 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.163551    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.166052 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.166021    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f486484c-dockercfg-mjgrc\""
Sep 04 23:04:51.166270 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.166255    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f486484c\""
Sep 04 23:04:51.173144 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.173116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:04:51.199396 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.199368    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:04:51.202579 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.202563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.205992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.205969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:04:51.212218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.212195    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:04:51.246650 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.246610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.246650 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.246645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.246905 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.246717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48lxs\" (UniqueName: \"kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.246905 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.246771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspxb\" (UniqueName: \"kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.348237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.348237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.348484 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48lxs\" (UniqueName: \"kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.348484 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspxb\" (UniqueName: \"kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.348962 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.349003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.348976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.356388 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.356359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48lxs\" (UniqueName: \"kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs\") pod \"payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.356512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.356402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspxb\" (UniqueName: \"kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb\") pod \"payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") " pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.474557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.474469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:51.513834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.513799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:51.745142 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.744962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:04:51.747173 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:04:51.747145    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14632cc_22b1_4321_81fe_a7c74df902c8.slice/crio-fc35d605c8555a436cd0af204bebc4b816e062a7bb5658e848bd57accfcddfc9 WatchSource:0}: Error finding container fc35d605c8555a436cd0af204bebc4b816e062a7bb5658e848bd57accfcddfc9: Status 404 returned error can't find the container with id fc35d605c8555a436cd0af204bebc4b816e062a7bb5658e848bd57accfcddfc9
Sep 04 23:04:51.805308 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:51.805126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:04:51.811876 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:04:51.811849    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81727017_8d85_47c3_85bc_d622b7fb3d68.slice/crio-23113359f4af020386bb47bb09840484a6aa74b27f7c169817dab7aa99c151c0 WatchSource:0}: Error finding container 23113359f4af020386bb47bb09840484a6aa74b27f7c169817dab7aa99c151c0: Status 404 returned error can't find the container with id 23113359f4af020386bb47bb09840484a6aa74b27f7c169817dab7aa99c151c0
Sep 04 23:04:52.080491 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.080397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" event={"ID":"81727017-8d85-47c3-85bc-d622b7fb3d68","Type":"ContainerStarted","Data":"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"}
Sep 04 23:04:52.080491 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.080451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" event={"ID":"81727017-8d85-47c3-85bc-d622b7fb3d68","Type":"ContainerStarted","Data":"23113359f4af020386bb47bb09840484a6aa74b27f7c169817dab7aa99c151c0"}
Sep 04 23:04:52.080760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.080505    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:04:52.082142 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.082117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" event={"ID":"c14632cc-22b1-4321-81fe-a7c74df902c8","Type":"ContainerStarted","Data":"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"}
Sep 04 23:04:52.082264 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.082170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" event={"ID":"c14632cc-22b1-4321-81fe-a7c74df902c8","Type":"ContainerStarted","Data":"fc35d605c8555a436cd0af204bebc4b816e062a7bb5658e848bd57accfcddfc9"}
Sep 04 23:04:52.082264 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.082199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:04:52.097178 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.097130    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" podStartSLOduration=1.097117846 podStartE2EDuration="1.097117846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:04:52.096692469 +0000 UTC m=+1176.267767079" watchObservedRunningTime="2026-09-04 23:04:52.097117846 +0000 UTC m=+1176.268192455"
Sep 04 23:04:52.113634 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:52.113576    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" podStartSLOduration=1.113555984 podStartE2EDuration="1.113555984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:04:52.111389315 +0000 UTC m=+1176.282463928" watchObservedRunningTime="2026-09-04 23:04:52.113555984 +0000 UTC m=+1176.284630601"
Sep 04 23:04:53.038341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:53.038308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:04:53.038751 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:04:53.038382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:05:03.090175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:03.090134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:05:03.090638 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:03.090220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:05:03.191203 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:03.191165    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:05:09.411101 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.411063    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:05:09.414554 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.414535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.417892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.417863    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-146c0e\""
Sep 04 23:05:09.418032 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.417938    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-146c0e-dockercfg-r8r49\""
Sep 04 23:05:09.431128 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.431091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:05:09.432771 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.432729    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:05:09.436328 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.436311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.447620 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.447598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:05:09.503214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.503175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78sh\" (UniqueName: \"kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.503214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.503228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.503434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.503347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.503434 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.503381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2gb\" (UniqueName: \"kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.604510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.604468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.604510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.604512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2gb\" (UniqueName: \"kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.604718 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.604544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78sh\" (UniqueName: \"kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.604718 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.604574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.605318 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.605297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.605362 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.605305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.613109 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.613084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78sh\" (UniqueName: \"kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh\") pod \"payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.613218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.613159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2gb\" (UniqueName: \"kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb\") pod \"payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.725195 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.725098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:09.745377 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.745348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:09.998254 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:09.998230    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:05:10.000172 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:05:10.000146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459f95d2_432b_4601_b4d8_1630e0431164.slice/crio-2ed96abb65f298b1ba9d4be6985e7884f6c3d3f3df482269016e9d87866c5681 WatchSource:0}: Error finding container 2ed96abb65f298b1ba9d4be6985e7884f6c3d3f3df482269016e9d87866c5681: Status 404 returned error can't find the container with id 2ed96abb65f298b1ba9d4be6985e7884f6c3d3f3df482269016e9d87866c5681
Sep 04 23:05:10.022303 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.022277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:05:10.025307 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:05:10.025284    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951b4db4_b28e_436f_8397_b97404d6477c.slice/crio-df2e6d609b44aad2d8dac7b224a84e717254da2580b633a5ca6e6850538c1692 WatchSource:0}: Error finding container df2e6d609b44aad2d8dac7b224a84e717254da2580b633a5ca6e6850538c1692: Status 404 returned error can't find the container with id df2e6d609b44aad2d8dac7b224a84e717254da2580b633a5ca6e6850538c1692
Sep 04 23:05:10.161637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.161601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" event={"ID":"951b4db4-b28e-436f-8397-b97404d6477c","Type":"ContainerStarted","Data":"c833cec0a22d9b5ed78a4798a1998f441c91a2cb18bb4e5cbbf92ac2a832a901"}
Sep 04 23:05:10.161857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.161704    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:10.161857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.161749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" event={"ID":"951b4db4-b28e-436f-8397-b97404d6477c","Type":"ContainerStarted","Data":"df2e6d609b44aad2d8dac7b224a84e717254da2580b633a5ca6e6850538c1692"}
Sep 04 23:05:10.163279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.163247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" event={"ID":"459f95d2-432b-4601-b4d8-1630e0431164","Type":"ContainerStarted","Data":"eec905c0d7e254ee8d59644397adcfd524f4a6f366ff8294b83b76b61eca95c4"}
Sep 04 23:05:10.163384 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.163296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" event={"ID":"459f95d2-432b-4601-b4d8-1630e0431164","Type":"ContainerStarted","Data":"2ed96abb65f298b1ba9d4be6985e7884f6c3d3f3df482269016e9d87866c5681"}
Sep 04 23:05:10.163384 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.163324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:10.177330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.177288    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" podStartSLOduration=1.177272919 podStartE2EDuration="1.177272919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:05:10.176422638 +0000 UTC m=+1194.347497251" watchObservedRunningTime="2026-09-04 23:05:10.177272919 +0000 UTC m=+1194.348347531"
Sep 04 23:05:10.191420 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:10.191373    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" podStartSLOduration=1.191354243 podStartE2EDuration="1.191354243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:05:10.191143272 +0000 UTC m=+1194.362217885" watchObservedRunningTime="2026-09-04 23:05:10.191354243 +0000 UTC m=+1194.362428854"
Sep 04 23:05:16.004397 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:16.004356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:05:21.170253 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:21.170216    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:05:21.170759 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:21.170720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:05:31.895048 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:31.895001    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:05:31.895472 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:31.895273    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" podUID="81727017-8d85-47c3-85bc-d622b7fb3d68" containerName="payload-processing" containerID="cri-o://e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740" gracePeriod=30
Sep 04 23:05:31.904624 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:31.904596    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:05:31.904894 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:31.904866    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" podUID="c14632cc-22b1-4321-81fe-a7c74df902c8" containerName="payload-pre-processing" containerID="cri-o://eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a" gracePeriod=30
Sep 04 23:05:32.164524 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.164496    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:05:32.167238 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.167213    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:05:32.205264 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume\") pod \"c14632cc-22b1-4321-81fe-a7c74df902c8\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") "
Sep 04 23:05:32.205423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume\") pod \"81727017-8d85-47c3-85bc-d622b7fb3d68\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") "
Sep 04 23:05:32.205423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fspxb\" (UniqueName: \"kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb\") pod \"81727017-8d85-47c3-85bc-d622b7fb3d68\" (UID: \"81727017-8d85-47c3-85bc-d622b7fb3d68\") "
Sep 04 23:05:32.205423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48lxs\" (UniqueName: \"kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs\") pod \"c14632cc-22b1-4321-81fe-a7c74df902c8\" (UID: \"c14632cc-22b1-4321-81fe-a7c74df902c8\") "
Sep 04 23:05:32.205686 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205640    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c14632cc-22b1-4321-81fe-a7c74df902c8" (UID: "c14632cc-22b1-4321-81fe-a7c74df902c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:05:32.205686 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.205672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81727017-8d85-47c3-85bc-d622b7fb3d68" (UID: "81727017-8d85-47c3-85bc-d622b7fb3d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:05:32.207499 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.207467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb" (OuterVolumeSpecName: "kube-api-access-fspxb") pod "81727017-8d85-47c3-85bc-d622b7fb3d68" (UID: "81727017-8d85-47c3-85bc-d622b7fb3d68"). InnerVolumeSpecName "kube-api-access-fspxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:05:32.207925 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.207904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs" (OuterVolumeSpecName: "kube-api-access-48lxs") pod "c14632cc-22b1-4321-81fe-a7c74df902c8" (UID: "c14632cc-22b1-4321-81fe-a7c74df902c8"). InnerVolumeSpecName "kube-api-access-48lxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:05:32.266011 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.265971    2577 generic.go:356] "Generic (PLEG): container finished" podID="81727017-8d85-47c3-85bc-d622b7fb3d68" containerID="e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740" exitCode=0
Sep 04 23:05:32.266190 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.266015    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"
Sep 04 23:05:32.266190 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.266054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" event={"ID":"81727017-8d85-47c3-85bc-d622b7fb3d68","Type":"ContainerDied","Data":"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"}
Sep 04 23:05:32.266190 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.266115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg" event={"ID":"81727017-8d85-47c3-85bc-d622b7fb3d68","Type":"ContainerDied","Data":"23113359f4af020386bb47bb09840484a6aa74b27f7c169817dab7aa99c151c0"}
Sep 04 23:05:32.266190 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.266129    2577 scope.go:117] "RemoveContainer" containerID="e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"
Sep 04 23:05:32.267507 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.267486    2577 generic.go:356] "Generic (PLEG): container finished" podID="c14632cc-22b1-4321-81fe-a7c74df902c8" containerID="eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a" exitCode=0
Sep 04 23:05:32.267619 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.267549    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"
Sep 04 23:05:32.267685 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.267553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" event={"ID":"c14632cc-22b1-4321-81fe-a7c74df902c8","Type":"ContainerDied","Data":"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"}
Sep 04 23:05:32.267761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.267685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284" event={"ID":"c14632cc-22b1-4321-81fe-a7c74df902c8","Type":"ContainerDied","Data":"fc35d605c8555a436cd0af204bebc4b816e062a7bb5658e848bd57accfcddfc9"}
Sep 04 23:05:32.275693 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.275679    2577 scope.go:117] "RemoveContainer" containerID="e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"
Sep 04 23:05:32.276013 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:05:32.275996    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740\": container with ID starting with e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740 not found: ID does not exist" containerID="e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"
Sep 04 23:05:32.276074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.276021    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740"} err="failed to get container status \"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740\": rpc error: code = NotFound desc = could not find container \"e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740\": container with ID starting with e1b16e7b08b0870801797766c06b5a6455f8cc7e51d5989567ea5dfd3e143740 not found: ID does not exist"
Sep 04 23:05:32.276074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.276038    2577 scope.go:117] "RemoveContainer" containerID="eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"
Sep 04 23:05:32.283409 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.283394    2577 scope.go:117] "RemoveContainer" containerID="eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"
Sep 04 23:05:32.283685 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:05:32.283667    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a\": container with ID starting with eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a not found: ID does not exist" containerID="eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"
Sep 04 23:05:32.283765 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.283693    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a"} err="failed to get container status \"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a\": rpc error: code = NotFound desc = could not find container \"eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a\": container with ID starting with eb40a2be1e15b1cb4df17d8c36363aa626050432a72b326f92dba17c9422b64a not found: ID does not exist"
Sep 04 23:05:32.293027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.290617    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:05:32.295372 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.295348    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f486484c-8b778f97c-d6sbg"]
Sep 04 23:05:32.305889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.305864    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81727017-8d85-47c3-85bc-d622b7fb3d68-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:05:32.305889 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.305890    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspxb\" (UniqueName: \"kubernetes.io/projected/81727017-8d85-47c3-85bc-d622b7fb3d68-kube-api-access-fspxb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:05:32.306020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.305895    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:05:32.306020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.305904    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48lxs\" (UniqueName: \"kubernetes.io/projected/c14632cc-22b1-4321-81fe-a7c74df902c8-kube-api-access-48lxs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:05:32.306020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.305917    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c14632cc-22b1-4321-81fe-a7c74df902c8-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:05:32.310116 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.310095    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f486484c-974f796b6-v4284"]
Sep 04 23:05:32.324708 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.324681    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81727017-8d85-47c3-85bc-d622b7fb3d68" path="/var/lib/kubelet/pods/81727017-8d85-47c3-85bc-d622b7fb3d68/volumes"
Sep 04 23:05:32.325052 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:32.325039    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14632cc-22b1-4321-81fe-a7c74df902c8" path="/var/lib/kubelet/pods/c14632cc-22b1-4321-81fe-a7c74df902c8/volumes"
Sep 04 23:05:36.187346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:36.187310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:05:45.696586 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.696546    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:05:45.697010 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.696992    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81727017-8d85-47c3-85bc-d622b7fb3d68" containerName="payload-processing"
Sep 04 23:05:45.697060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.697013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81727017-8d85-47c3-85bc-d622b7fb3d68" containerName="payload-processing"
Sep 04 23:05:45.697060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.697033    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14632cc-22b1-4321-81fe-a7c74df902c8" containerName="payload-pre-processing"
Sep 04 23:05:45.697060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.697041    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14632cc-22b1-4321-81fe-a7c74df902c8" containerName="payload-pre-processing"
Sep 04 23:05:45.697157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.697112    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14632cc-22b1-4321-81fe-a7c74df902c8" containerName="payload-pre-processing"
Sep 04 23:05:45.697157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.697127    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81727017-8d85-47c3-85bc-d622b7fb3d68" containerName="payload-processing"
Sep 04 23:05:45.699970 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.699953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.703461 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.703439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"kube-root-ca.crt\""
Sep 04 23:05:45.703600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.703437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"default-dockercfg-ql7bt\""
Sep 04 23:05:45.703600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.703486    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"openshift-service-ca.crt\""
Sep 04 23:05:45.703600 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.703577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"auth-test-model-9ec1e5-kserve-self-signed-certs\""
Sep 04 23:05:45.705500 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.705625 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.705625 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.705707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705639    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.705770 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.705770 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.705752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pdf\" (UniqueName: \"kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.708692 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.708671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:05:45.806823 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.806992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.806992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.806992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pdf\" (UniqueName: \"kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.806992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.806992 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.806981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.807322 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.807295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.807385 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.807334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.807443 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.807423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.809063 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.809045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.809309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.809289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:45.814570 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:45.814553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pdf\" (UniqueName: \"kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf\") pod \"auth-test-model-9ec1e5-kserve-84f989d6c-62qxs\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:46.010070 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:46.009948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:46.279712 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:46.279636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:05:46.282625 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:05:46.282599    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bf4c1e4_11fc_4149_b733_aed3ff42e9f4.slice/crio-2dc775ea234676b121af6e5337885833096c15df1531737945eb93c5b558f348 WatchSource:0}: Error finding container 2dc775ea234676b121af6e5337885833096c15df1531737945eb93c5b558f348: Status 404 returned error can't find the container with id 2dc775ea234676b121af6e5337885833096c15df1531737945eb93c5b558f348
Sep 04 23:05:46.332338 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:46.332310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" event={"ID":"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4","Type":"ContainerStarted","Data":"2dc775ea234676b121af6e5337885833096c15df1531737945eb93c5b558f348"}
Sep 04 23:05:47.338147 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:47.338113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" event={"ID":"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4","Type":"ContainerStarted","Data":"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"}
Sep 04 23:05:47.359418 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:47.359366    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" podStartSLOduration=2.35935352 podStartE2EDuration="2.35935352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:05:47.357951652 +0000 UTC m=+1231.529026267" watchObservedRunningTime="2026-09-04 23:05:47.35935352 +0000 UTC m=+1231.530428132"
Sep 04 23:05:47.895648 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:47.895612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:05:56.011088 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:56.011052    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:56.011557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:56.011135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:56.023487 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:56.023463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:56.387207 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:56.387169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:05:57.705463 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.705427    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:05:57.709483 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.709464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.712040 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.712016    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ed43d3\""
Sep 04 23:05:57.712155 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.712083    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ed43d3-dockercfg-x5vcx\""
Sep 04 23:05:57.716147 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.716126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:05:57.802587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.802551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.802587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.802588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fvq\" (UniqueName: \"kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.903522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.903487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.903522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.903523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fvq\" (UniqueName: \"kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.904200 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.904180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:57.911236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:57.911212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fvq\" (UniqueName: \"kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq\") pod \"payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:58.020528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:58.020444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:05:58.289660 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:58.289637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:05:58.292212 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:05:58.292186    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bb34c3_68fe_45e4_bd0c_816b329d87af.slice/crio-1a335503e0afe2888f3339dde356694524d5155dcf20df2f70b4ba299db433f6 WatchSource:0}: Error finding container 1a335503e0afe2888f3339dde356694524d5155dcf20df2f70b4ba299db433f6: Status 404 returned error can't find the container with id 1a335503e0afe2888f3339dde356694524d5155dcf20df2f70b4ba299db433f6
Sep 04 23:05:58.388248 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:58.388215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" event={"ID":"77bb34c3-68fe-45e4-bd0c-816b329d87af","Type":"ContainerStarted","Data":"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"}
Sep 04 23:05:58.388427 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:58.388267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" event={"ID":"77bb34c3-68fe-45e4-bd0c-816b329d87af","Type":"ContainerStarted","Data":"1a335503e0afe2888f3339dde356694524d5155dcf20df2f70b4ba299db433f6"}
Sep 04 23:05:58.403620 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:58.403514    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" podStartSLOduration=1.403495203 podStartE2EDuration="1.403495203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:05:58.402823265 +0000 UTC m=+1242.573897876" watchObservedRunningTime="2026-09-04 23:05:58.403495203 +0000 UTC m=+1242.574569815"
Sep 04 23:05:59.392658 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:05:59.392618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:06:02.799527 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:02.799486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:06:10.399906 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:10.399833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:06:20.686394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.686359    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:06:20.690965 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.690943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.694557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.694532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"default-dockercfg-qq6dq\""
Sep 04 23:06:20.694758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.694533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"kube-root-ca.crt\""
Sep 04 23:06:20.694758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.694532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"auth-test-model-146c0e-kserve-self-signed-certs\""
Sep 04 23:06:20.694758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.694538    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"openshift-service-ca.crt\""
Sep 04 23:06:20.700632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.700602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:06:20.796025 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.795988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.796025 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.796030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.796259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.796058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8xx\" (UniqueName: \"kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.796259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.796135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.796259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.796180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.796259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.796203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.896869 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.896832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.896869 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.896873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897064 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.896973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897064 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897064 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8xx\" (UniqueName: \"kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897315 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897432 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897374    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.897432 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.897403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.899268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.899238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.899368 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.899349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:20.904594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:20.904576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8xx\" (UniqueName: \"kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx\") pod \"auth-test-model-146c0e-kserve-5449d948c5-w8j8x\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:21.001892 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:21.001801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:21.272852 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:21.272760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:06:21.276981 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:06:21.276953    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89b098a1_b16e_47dc_b0de_06e8a2c27c1b.slice/crio-ded3509da206230bdfdbe0c0779ebd24c85ec178bca96e4296adf64001fa2e38 WatchSource:0}: Error finding container ded3509da206230bdfdbe0c0779ebd24c85ec178bca96e4296adf64001fa2e38: Status 404 returned error can't find the container with id ded3509da206230bdfdbe0c0779ebd24c85ec178bca96e4296adf64001fa2e38
Sep 04 23:06:21.490091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:21.490043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" event={"ID":"89b098a1-b16e-47dc-b0de-06e8a2c27c1b","Type":"ContainerStarted","Data":"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"}
Sep 04 23:06:21.490091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:21.490095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" event={"ID":"89b098a1-b16e-47dc-b0de-06e8a2c27c1b","Type":"ContainerStarted","Data":"ded3509da206230bdfdbe0c0779ebd24c85ec178bca96e4296adf64001fa2e38"}
Sep 04 23:06:21.511005 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:21.510958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" podStartSLOduration=1.510944573 podStartE2EDuration="1.510944573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:06:21.510140797 +0000 UTC m=+1265.681215411" watchObservedRunningTime="2026-09-04 23:06:21.510944573 +0000 UTC m=+1265.682019185"
Sep 04 23:06:26.699054 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.699018    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:06:26.867263 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.867222    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:06:26.870589 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.870567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:26.873210 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.873181    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-217ad6\""
Sep 04 23:06:26.873317 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.873267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-217ad6-dockercfg-wmdrr\""
Sep 04 23:06:26.886454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.886420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:06:26.953475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.953395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:26.953475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:26.953432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8t8\" (UniqueName: \"kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.054704 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.054665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.054704 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.054709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8t8\" (UniqueName: \"kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.055357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.055337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.066574 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.066541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8t8\" (UniqueName: \"kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8\") pod \"payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.182596 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.182320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:27.478373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.478328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:06:27.480780 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:06:27.480729    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036d6b92_1f51_4bd4_86c6_8c20c9601e38.slice/crio-9c37cf20963a98a043af6c2db10eaf8c23c786d57bc65a6299df74a53d9bba94 WatchSource:0}: Error finding container 9c37cf20963a98a043af6c2db10eaf8c23c786d57bc65a6299df74a53d9bba94: Status 404 returned error can't find the container with id 9c37cf20963a98a043af6c2db10eaf8c23c786d57bc65a6299df74a53d9bba94
Sep 04 23:06:27.517388 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:27.517355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" event={"ID":"036d6b92-1f51-4bd4-86c6-8c20c9601e38","Type":"ContainerStarted","Data":"9c37cf20963a98a043af6c2db10eaf8c23c786d57bc65a6299df74a53d9bba94"}
Sep 04 23:06:28.523818 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:28.523781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" event={"ID":"036d6b92-1f51-4bd4-86c6-8c20c9601e38","Type":"ContainerStarted","Data":"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"}
Sep 04 23:06:28.524297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:28.523862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:28.541667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:28.541620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" podStartSLOduration=2.541605321 podStartE2EDuration="2.541605321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:06:28.541211862 +0000 UTC m=+1272.712286474" watchObservedRunningTime="2026-09-04 23:06:28.541605321 +0000 UTC m=+1272.712679946"
Sep 04 23:06:31.002290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:31.002249    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:31.002813 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:31.002307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:31.014612 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:31.014589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:31.549041 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:31.549009    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:06:39.530343 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:39.530305    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:06:43.794665 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:43.794629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:06:50.878124 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:50.878092    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:50.881331 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:50.881314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:50.883953 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:50.883928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s76bh\""
Sep 04 23:06:50.888943 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:50.888922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:50.965535 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:50.965491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2lf\" (UniqueName: \"kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf\") pod \"test-curl-7999-139848597566784\" (UID: \"4a263063-34b8-4862-b83e-95a8d71281d0\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:51.066470 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.066433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2lf\" (UniqueName: \"kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf\") pod \"test-curl-7999-139848597566784\" (UID: \"4a263063-34b8-4862-b83e-95a8d71281d0\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:51.074437 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.074415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2lf\" (UniqueName: \"kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf\") pod \"test-curl-7999-139848597566784\" (UID: \"4a263063-34b8-4862-b83e-95a8d71281d0\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:51.191486 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.191398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:51.458100 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.458013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:51.461171 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:06:51.461145    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a263063_34b8_4862_b83e_95a8d71281d0.slice/crio-a12981b2cfad034a5ea7ac9c9c56d02b30c985da048f5b3af6037bb9cd206a79 WatchSource:0}: Error finding container a12981b2cfad034a5ea7ac9c9c56d02b30c985da048f5b3af6037bb9cd206a79: Status 404 returned error can't find the container with id a12981b2cfad034a5ea7ac9c9c56d02b30c985da048f5b3af6037bb9cd206a79
Sep 04 23:06:51.462837 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.462819    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:06:51.628907 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:51.628873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597566784" event={"ID":"4a263063-34b8-4862-b83e-95a8d71281d0","Type":"ContainerStarted","Data":"a12981b2cfad034a5ea7ac9c9c56d02b30c985da048f5b3af6037bb9cd206a79"}
Sep 04 23:06:52.635151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:52.635112    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a263063-34b8-4862-b83e-95a8d71281d0" containerID="d0d906a6cdfb9072fe45e19bb2ae54e4e4560298835767e1cae35c4f357d1245" exitCode=0
Sep 04 23:06:52.635537 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:52.635159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597566784" event={"ID":"4a263063-34b8-4862-b83e-95a8d71281d0","Type":"ContainerDied","Data":"d0d906a6cdfb9072fe45e19bb2ae54e4e4560298835767e1cae35c4f357d1245"}
Sep 04 23:06:53.768145 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.768120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:53.802232 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.802208    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7999-139848597566784_4a263063-34b8-4862-b83e-95a8d71281d0/test-curl-7999-139848597566784/0.log"
Sep 04 23:06:53.825967 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.825939    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:53.829566 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.829543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:53.888370 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.888339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r2lf\" (UniqueName: \"kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf\") pod \"4a263063-34b8-4862-b83e-95a8d71281d0\" (UID: \"4a263063-34b8-4862-b83e-95a8d71281d0\") "
Sep 04 23:06:53.890511 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.890489    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf" (OuterVolumeSpecName: "kube-api-access-4r2lf") pod "4a263063-34b8-4862-b83e-95a8d71281d0" (UID: "4a263063-34b8-4862-b83e-95a8d71281d0"). InnerVolumeSpecName "kube-api-access-4r2lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:06:53.962921 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.962888    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:53.963237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.963224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a263063-34b8-4862-b83e-95a8d71281d0" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:53.963287 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.963238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a263063-34b8-4862-b83e-95a8d71281d0" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:53.963324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.963304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a263063-34b8-4862-b83e-95a8d71281d0" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:53.966106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.966089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:53.971680 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.971654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:53.989523 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:53.989495    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r2lf\" (UniqueName: \"kubernetes.io/projected/4a263063-34b8-4862-b83e-95a8d71281d0-kube-api-access-4r2lf\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:06:54.090595 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.090562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5wb\" (UniqueName: \"kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb\") pod \"test-curl-7999-139848597566784\" (UID: \"81662b7e-5ccb-4851-b345-3822fe379214\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:54.191606 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.191522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5wb\" (UniqueName: \"kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb\") pod \"test-curl-7999-139848597566784\" (UID: \"81662b7e-5ccb-4851-b345-3822fe379214\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:54.200606 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.200585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5wb\" (UniqueName: \"kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb\") pod \"test-curl-7999-139848597566784\" (UID: \"81662b7e-5ccb-4851-b345-3822fe379214\") " pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:54.275872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.275842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:54.328110 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.328063    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a263063-34b8-4862-b83e-95a8d71281d0" path="/var/lib/kubelet/pods/4a263063-34b8-4862-b83e-95a8d71281d0/volumes"
Sep 04 23:06:54.543957 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.543934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:54.546298 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:06:54.546265    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81662b7e_5ccb_4851_b345_3822fe379214.slice/crio-d35860d4ec4044fe4c11f5659c4e72d3e7cb69f134e6d77a2d5974a840bde0eb WatchSource:0}: Error finding container d35860d4ec4044fe4c11f5659c4e72d3e7cb69f134e6d77a2d5974a840bde0eb: Status 404 returned error can't find the container with id d35860d4ec4044fe4c11f5659c4e72d3e7cb69f134e6d77a2d5974a840bde0eb
Sep 04 23:06:54.645982 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.645941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597566784" event={"ID":"81662b7e-5ccb-4851-b345-3822fe379214","Type":"ContainerStarted","Data":"d35860d4ec4044fe4c11f5659c4e72d3e7cb69f134e6d77a2d5974a840bde0eb"}
Sep 04 23:06:54.647163 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.647141    2577 scope.go:117] "RemoveContainer" containerID="d0d906a6cdfb9072fe45e19bb2ae54e4e4560298835767e1cae35c4f357d1245"
Sep 04 23:06:54.647295 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:54.647146    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:55.653362 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:55.653326    2577 generic.go:356] "Generic (PLEG): container finished" podID="81662b7e-5ccb-4851-b345-3822fe379214" containerID="9952799e931ad53ca6120d528bc8fe6ccce4010778c80806e1d7c3bdb8402dc4" exitCode=0
Sep 04 23:06:55.653800 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:55.653414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597566784" event={"ID":"81662b7e-5ccb-4851-b345-3822fe379214","Type":"ContainerDied","Data":"9952799e931ad53ca6120d528bc8fe6ccce4010778c80806e1d7c3bdb8402dc4"}
Sep 04 23:06:56.790899 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.790874    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:56.810901 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.810880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7999-139848597566784_81662b7e-5ccb-4851-b345-3822fe379214/test-curl-7999-139848597566784/0.log"
Sep 04 23:06:56.833616 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.833591    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:56.836999 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.836978    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7999-139848597566784"]
Sep 04 23:06:56.913587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.913498    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v5wb\" (UniqueName: \"kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb\") pod \"81662b7e-5ccb-4851-b345-3822fe379214\" (UID: \"81662b7e-5ccb-4851-b345-3822fe379214\") "
Sep 04 23:06:56.915628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.915598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb" (OuterVolumeSpecName: "kube-api-access-5v5wb") pod "81662b7e-5ccb-4851-b345-3822fe379214" (UID: "81662b7e-5ccb-4851-b345-3822fe379214"). InnerVolumeSpecName "kube-api-access-5v5wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:06:56.960085 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.960059    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:56.960401 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.960389    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81662b7e-5ccb-4851-b345-3822fe379214" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:56.960570 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.960403    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81662b7e-5ccb-4851-b345-3822fe379214" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:56.960570 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.960457    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81662b7e-5ccb-4851-b345-3822fe379214" containerName="test-curl-7999-139848597566784"
Sep 04 23:06:56.963175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.963160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:56.967421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:56.967398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:57.014577 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.014547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8djx\" (UniqueName: \"kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx\") pod \"test-curl-7999-139848597567360\" (UID: \"b5e63747-ddf9-47f2-be81-6870f0b97f79\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:57.014695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.014611    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5wb\" (UniqueName: \"kubernetes.io/projected/81662b7e-5ccb-4851-b345-3822fe379214-kube-api-access-5v5wb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:06:57.115220 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.115189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8djx\" (UniqueName: \"kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx\") pod \"test-curl-7999-139848597567360\" (UID: \"b5e63747-ddf9-47f2-be81-6870f0b97f79\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:57.123142 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.123113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8djx\" (UniqueName: \"kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx\") pod \"test-curl-7999-139848597567360\" (UID: \"b5e63747-ddf9-47f2-be81-6870f0b97f79\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:57.273481 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.273398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:57.544473 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.544390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:57.546921 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:06:57.546896    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5e63747_ddf9_47f2_be81_6870f0b97f79.slice/crio-04991f531747e4e9d8468dc8ec94e4caca7fea9fb278b31bf16aa2db6b9d5d24 WatchSource:0}: Error finding container 04991f531747e4e9d8468dc8ec94e4caca7fea9fb278b31bf16aa2db6b9d5d24: Status 404 returned error can't find the container with id 04991f531747e4e9d8468dc8ec94e4caca7fea9fb278b31bf16aa2db6b9d5d24
Sep 04 23:06:57.664890 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.664859    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:57.664890 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.664873    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35860d4ec4044fe4c11f5659c4e72d3e7cb69f134e6d77a2d5974a840bde0eb"
Sep 04 23:06:57.666136 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:57.666110    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597567360" event={"ID":"b5e63747-ddf9-47f2-be81-6870f0b97f79","Type":"ContainerStarted","Data":"04991f531747e4e9d8468dc8ec94e4caca7fea9fb278b31bf16aa2db6b9d5d24"}
Sep 04 23:06:57.675404 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:06:57.675379    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7999-139848597566784\" is forbidden: User \"system:node:ip-10-0-134-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-172.ec2.internal' and this object" podUID="81662b7e-5ccb-4851-b345-3822fe379214" pod="opendatahub/test-curl-7999-139848597566784"
Sep 04 23:06:58.330176 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:58.326500    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81662b7e-5ccb-4851-b345-3822fe379214" path="/var/lib/kubelet/pods/81662b7e-5ccb-4851-b345-3822fe379214/volumes"
Sep 04 23:06:58.672227 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:58.672184    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5e63747-ddf9-47f2-be81-6870f0b97f79" containerID="4ed7e679c8ce46335b92ce8821123c2a38982d87a2d3d5fe85597b495eef2ee4" exitCode=0
Sep 04 23:06:58.672408 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:58.672269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597567360" event={"ID":"b5e63747-ddf9-47f2-be81-6870f0b97f79","Type":"ContainerDied","Data":"4ed7e679c8ce46335b92ce8821123c2a38982d87a2d3d5fe85597b495eef2ee4"}
Sep 04 23:06:59.804516 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.804494    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:59.823778 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.823747    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7999-139848597567360_b5e63747-ddf9-47f2-be81-6870f0b97f79/test-curl-7999-139848597567360/0.log"
Sep 04 23:06:59.845090 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.845062    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:59.850386 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.850365    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:59.942156 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.942084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8djx\" (UniqueName: \"kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx\") pod \"b5e63747-ddf9-47f2-be81-6870f0b97f79\" (UID: \"b5e63747-ddf9-47f2-be81-6870f0b97f79\") "
Sep 04 23:06:59.944133 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.944112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx" (OuterVolumeSpecName: "kube-api-access-z8djx") pod "b5e63747-ddf9-47f2-be81-6870f0b97f79" (UID: "b5e63747-ddf9-47f2-be81-6870f0b97f79"). InnerVolumeSpecName "kube-api-access-z8djx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:06:59.960798 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.960772    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:06:59.961157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.961140    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5e63747-ddf9-47f2-be81-6870f0b97f79" containerName="test-curl-7999-139848597567360"
Sep 04 23:06:59.961234 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.961160    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5e63747-ddf9-47f2-be81-6870f0b97f79" containerName="test-curl-7999-139848597567360"
Sep 04 23:06:59.961296 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.961233    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5e63747-ddf9-47f2-be81-6870f0b97f79" containerName="test-curl-7999-139848597567360"
Sep 04 23:06:59.964044 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.964027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:06:59.967538 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:06:59.967516    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:07:00.043322 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.043288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjct\" (UniqueName: \"kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct\") pod \"test-curl-7999-139848597567360\" (UID: \"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:00.043504 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.043355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8djx\" (UniqueName: \"kubernetes.io/projected/b5e63747-ddf9-47f2-be81-6870f0b97f79-kube-api-access-z8djx\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:07:00.144122 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.144082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjct\" (UniqueName: \"kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct\") pod \"test-curl-7999-139848597567360\" (UID: \"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:00.151583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.151562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjct\" (UniqueName: \"kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct\") pod \"test-curl-7999-139848597567360\" (UID: \"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1\") " pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:00.273398 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.273304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:00.328146 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.328111    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5e63747-ddf9-47f2-be81-6870f0b97f79" path="/var/lib/kubelet/pods/b5e63747-ddf9-47f2-be81-6870f0b97f79/volumes"
Sep 04 23:07:00.544370 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.544346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:07:00.545854 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:07:00.545831    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba30ae8b_222b_4ed6_8adb_8c5fe897d0c1.slice/crio-354eea821e4140a0b65fa9c8b2257aaf1abda6434111b19d3b46fe8adf1675fd WatchSource:0}: Error finding container 354eea821e4140a0b65fa9c8b2257aaf1abda6434111b19d3b46fe8adf1675fd: Status 404 returned error can't find the container with id 354eea821e4140a0b65fa9c8b2257aaf1abda6434111b19d3b46fe8adf1675fd
Sep 04 23:07:00.682777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.682728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:00.682777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.682780    2577 scope.go:117] "RemoveContainer" containerID="4ed7e679c8ce46335b92ce8821123c2a38982d87a2d3d5fe85597b495eef2ee4"
Sep 04 23:07:00.684133 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:00.684099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597567360" event={"ID":"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1","Type":"ContainerStarted","Data":"354eea821e4140a0b65fa9c8b2257aaf1abda6434111b19d3b46fe8adf1675fd"}
Sep 04 23:07:01.691448 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:01.691413    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" containerID="c1396e6624b5401c948887baf3351e5941bda9ede5f5e399137e3f18ec39bd2f" exitCode=0
Sep 04 23:07:01.691917 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:01.691495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7999-139848597567360" event={"ID":"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1","Type":"ContainerDied","Data":"c1396e6624b5401c948887baf3351e5941bda9ede5f5e399137e3f18ec39bd2f"}
Sep 04 23:07:02.828087 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.828059    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:02.847781 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.847750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7999-139848597567360_ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1/test-curl-7999-139848597567360/0.log"
Sep 04 23:07:02.871482 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.871455    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:07:02.875575 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.875552    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7999-139848597567360"]
Sep 04 23:07:02.965222 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.965135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hjct\" (UniqueName: \"kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct\") pod \"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1\" (UID: \"ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1\") "
Sep 04 23:07:02.967213 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:02.967175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct" (OuterVolumeSpecName: "kube-api-access-8hjct") pod "ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" (UID: "ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1"). InnerVolumeSpecName "kube-api-access-8hjct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:07:03.066399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:03.066360    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjct\" (UniqueName: \"kubernetes.io/projected/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1-kube-api-access-8hjct\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:07:03.303048 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:03.302965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:07:03.701569 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:03.701537    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:03.701720 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:03.701578    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="354eea821e4140a0b65fa9c8b2257aaf1abda6434111b19d3b46fe8adf1675fd"
Sep 04 23:07:03.712854 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:07:03.712825    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7999-139848597567360\" is forbidden: User \"system:node:ip-10-0-134-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-172.ec2.internal' and this object" podUID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" pod="opendatahub/test-curl-7999-139848597567360"
Sep 04 23:07:04.325802 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:04.325767    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" path="/var/lib/kubelet/pods/ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1/volumes"
Sep 04 23:07:09.327756 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.327704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:07:09.328101 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.328018    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" containerName="test-curl-7999-139848597567360"
Sep 04 23:07:09.328101 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.328030    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" containerName="test-curl-7999-139848597567360"
Sep 04 23:07:09.328101 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.328096    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba30ae8b-222b-4ed6-8adb-8c5fe897d0c1" containerName="test-curl-7999-139848597567360"
Sep 04 23:07:09.331245 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.331230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.333783 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.333757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9f64c8d3\""
Sep 04 23:07:09.333900 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.333825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9f64c8d3-dockercfg-nq68c\""
Sep 04 23:07:09.350102 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.350076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:07:09.420989 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.420947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztggc\" (UniqueName: \"kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.421248 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.421219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.522046 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.522013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.522046 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.522055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztggc\" (UniqueName: \"kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.522707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.522684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.529863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.529841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztggc\" (UniqueName: \"kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc\") pod \"payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.641068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.641035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:09.912827 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:09.912796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:07:09.914830 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:07:09.914802    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ccf919_b9df_42e1_a859_6ae9dae2645c.slice/crio-49e0d865238f2fcc4713edaeef7a0b6a99192bc23fc559dc85282cf124fb3e43 WatchSource:0}: Error finding container 49e0d865238f2fcc4713edaeef7a0b6a99192bc23fc559dc85282cf124fb3e43: Status 404 returned error can't find the container with id 49e0d865238f2fcc4713edaeef7a0b6a99192bc23fc559dc85282cf124fb3e43
Sep 04 23:07:10.734745 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:10.734704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" event={"ID":"52ccf919-b9df-42e1-a859-6ae9dae2645c","Type":"ContainerStarted","Data":"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"}
Sep 04 23:07:10.735214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:10.734790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:10.735214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:10.734810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" event={"ID":"52ccf919-b9df-42e1-a859-6ae9dae2645c","Type":"ContainerStarted","Data":"49e0d865238f2fcc4713edaeef7a0b6a99192bc23fc559dc85282cf124fb3e43"}
Sep 04 23:07:10.752519 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:10.752469    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" podStartSLOduration=1.752453494 podStartE2EDuration="1.752453494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:07:10.750997334 +0000 UTC m=+1314.922071947" watchObservedRunningTime="2026-09-04 23:07:10.752453494 +0000 UTC m=+1314.923528106"
Sep 04 23:07:16.440798 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:16.440767    2577 scope.go:117] "RemoveContainer" containerID="e0490ab3e75b97aad5e23bffc48fd4e42ca50c7aaa930cf5582a54128f46cae6"
Sep 04 23:07:16.448935 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:16.448917    2577 scope.go:117] "RemoveContainer" containerID="a92874dc4e850f001701880ddae7762b71aa0a4298920723fddc2de3b363f8a2"
Sep 04 23:07:16.456025 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:16.456009    2577 scope.go:117] "RemoveContainer" containerID="8a6b5c39e3fad9e7e96a4e3e22958185357a5eed43b6e0d703c9f593905d034f"
Sep 04 23:07:20.099344 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:20.099295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:07:21.742237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:21.742208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:07:37.891549 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:37.891517    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:07:54.789104 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:07:54.789067    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:08:08.586204 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:08.586166    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:08:21.344104 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.344071    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:08:21.346454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.346435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.351836 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.351814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f64c8d3\"/\"kube-root-ca.crt\""
Sep 04 23:08:21.351947 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.351835    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f64c8d3\"/\"e2e-lifecycle-model-9f64c8d3-kserve-self-signed-certs\""
Sep 04 23:08:21.352214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.352195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f64c8d3\"/\"default-dockercfg-r7sjd\""
Sep 04 23:08:21.352363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.352347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f64c8d3\"/\"openshift-service-ca.crt\""
Sep 04 23:08:21.379072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.379050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:08:21.456470 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.456470 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.456654 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng527\" (UniqueName: \"kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.456654 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.456654 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.456654 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.456616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557063 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng527\" (UniqueName: \"kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557241 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557576 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.557576 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.557566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.559285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.559264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.559652 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.559636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.565065 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.565048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng527\" (UniqueName: \"kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527\") pod \"e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") " pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.655536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.655453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:21.926042 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:21.926018    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:08:21.928460 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:08:21.928434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5e8e048_a19d_4787_b29e_b00fffa9bd9b.slice/crio-573cb458fd2023cce7406fc30dd091847ff138872d10ca30c6a85f56412895f1 WatchSource:0}: Error finding container 573cb458fd2023cce7406fc30dd091847ff138872d10ca30c6a85f56412895f1: Status 404 returned error can't find the container with id 573cb458fd2023cce7406fc30dd091847ff138872d10ca30c6a85f56412895f1
Sep 04 23:08:22.060911 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:22.060876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" event={"ID":"e5e8e048-a19d-4787-b29e-b00fffa9bd9b","Type":"ContainerStarted","Data":"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"}
Sep 04 23:08:22.061087 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:22.060923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" event={"ID":"e5e8e048-a19d-4787-b29e-b00fffa9bd9b","Type":"ContainerStarted","Data":"573cb458fd2023cce7406fc30dd091847ff138872d10ca30c6a85f56412895f1"}
Sep 04 23:08:22.084555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:22.084500    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" podStartSLOduration=1.084484585 podStartE2EDuration="1.084484585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:08:22.082425487 +0000 UTC m=+1386.253500112" watchObservedRunningTime="2026-09-04 23:08:22.084484585 +0000 UTC m=+1386.255559197"
Sep 04 23:08:25.799302 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:25.799265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:08:31.655899 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:31.655859    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:31.656373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:31.655915    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:31.668204 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:31.668182    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:32.116237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:32.116202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:08:38.752955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.752913    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8002-f45970"]
Sep 04 23:08:38.756570 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.756548    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:38.758849 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.758829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s76bh\""
Sep 04 23:08:38.762587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.762566    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8002-f45970"]
Sep 04 23:08:38.807671 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.807642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2m9\" (UniqueName: \"kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9\") pod \"mt-curl-8002-f45970\" (UID: \"5cfffe6e-e232-4cd8-a253-515ce76bb559\") " pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:38.908459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.908417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2m9\" (UniqueName: \"kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9\") pod \"mt-curl-8002-f45970\" (UID: \"5cfffe6e-e232-4cd8-a253-515ce76bb559\") " pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:38.916361 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:38.916327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2m9\" (UniqueName: \"kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9\") pod \"mt-curl-8002-f45970\" (UID: \"5cfffe6e-e232-4cd8-a253-515ce76bb559\") " pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:39.067396 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:39.067309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:39.332490 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:39.332460    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8002-f45970"]
Sep 04 23:08:39.335776 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:08:39.335751    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cfffe6e_e232_4cd8_a253_515ce76bb559.slice/crio-7ccf4321e47f112d32461230a082982366608908b09b44f467ac0b940266b01a WatchSource:0}: Error finding container 7ccf4321e47f112d32461230a082982366608908b09b44f467ac0b940266b01a: Status 404 returned error can't find the container with id 7ccf4321e47f112d32461230a082982366608908b09b44f467ac0b940266b01a
Sep 04 23:08:40.142418 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:40.142389    2577 generic.go:356] "Generic (PLEG): container finished" podID="5cfffe6e-e232-4cd8-a253-515ce76bb559" containerID="1e8b02e73e9e6b005039f4ace24f61066b5b6751714c1f6a5f016854d9909464" exitCode=0
Sep 04 23:08:40.142728 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:40.142448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8002-f45970" event={"ID":"5cfffe6e-e232-4cd8-a253-515ce76bb559","Type":"ContainerDied","Data":"1e8b02e73e9e6b005039f4ace24f61066b5b6751714c1f6a5f016854d9909464"}
Sep 04 23:08:40.142728 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:40.142487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8002-f45970" event={"ID":"5cfffe6e-e232-4cd8-a253-515ce76bb559","Type":"ContainerStarted","Data":"7ccf4321e47f112d32461230a082982366608908b09b44f467ac0b940266b01a"}
Sep 04 23:08:41.280076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.280050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:41.311250 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.311216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8002-f45970_5cfffe6e-e232-4cd8-a253-515ce76bb559/mt-curl-8002-f45970/0.log"
Sep 04 23:08:41.332428 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.332395    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8002-f45970"]
Sep 04 23:08:41.334540 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.334509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh2m9\" (UniqueName: \"kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9\") pod \"5cfffe6e-e232-4cd8-a253-515ce76bb559\" (UID: \"5cfffe6e-e232-4cd8-a253-515ce76bb559\") "
Sep 04 23:08:41.336094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.336074    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8002-f45970"]
Sep 04 23:08:41.336701 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.336676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9" (OuterVolumeSpecName: "kube-api-access-dh2m9") pod "5cfffe6e-e232-4cd8-a253-515ce76bb559" (UID: "5cfffe6e-e232-4cd8-a253-515ce76bb559"). InnerVolumeSpecName "kube-api-access-dh2m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:08:41.435639 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:41.435547    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2m9\" (UniqueName: \"kubernetes.io/projected/5cfffe6e-e232-4cd8-a253-515ce76bb559-kube-api-access-dh2m9\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:42.153957 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:42.153924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:42.153957 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:42.153924    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ccf4321e47f112d32461230a082982366608908b09b44f467ac0b940266b01a"
Sep 04 23:08:42.163754 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:08:42.163710    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-8002-f45970\" is forbidden: User \"system:node:ip-10-0-134-172.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-172.ec2.internal' and this object" podUID="5cfffe6e-e232-4cd8-a253-515ce76bb559" pod="opendatahub/mt-curl-8002-f45970"
Sep 04 23:08:42.326365 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:42.326331    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cfffe6e-e232-4cd8-a253-515ce76bb559" path="/var/lib/kubelet/pods/5cfffe6e-e232-4cd8-a253-515ce76bb559/volumes"
Sep 04 23:08:42.686998 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:42.686966    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:08:43.185747 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:43.185703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:08:43.783221 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:43.783188    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:08:55.988301 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:55.988264    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:08:57.706246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:57.706207    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" podUID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" containerName="main" containerID="cri-o://67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990" gracePeriod=45
Sep 04 23:08:57.945447 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:57.945423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:08:58.077551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077457    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077644    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077661    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4pdf\" (UniqueName: \"kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf\") pod \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\" (UID: \"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4\") "
Sep 04 23:08:58.077879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:58.078037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home" (OuterVolumeSpecName: "home") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:58.078037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:58.078037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.077996    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.078037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.078013    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.078037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.078026    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.079848 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.079819    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm" (OuterVolumeSpecName: "dshm") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:58.079954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.079861    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf" (OuterVolumeSpecName: "kube-api-access-k4pdf") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "kube-api-access-k4pdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:08:58.079954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.079906    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" (UID: "7bf4c1e4-11fc-4149-b733-aed3ff42e9f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:08:58.178479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.178431    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.178479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.178476    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pdf\" (UniqueName: \"kubernetes.io/projected/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-kube-api-access-k4pdf\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.178479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.178487    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:58.233094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.233060    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" containerID="67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990" exitCode=0
Sep 04 23:08:58.233230 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.233107    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"
Sep 04 23:08:58.233230 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.233117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" event={"ID":"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4","Type":"ContainerDied","Data":"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"}
Sep 04 23:08:58.233230 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.233150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs" event={"ID":"7bf4c1e4-11fc-4149-b733-aed3ff42e9f4","Type":"ContainerDied","Data":"2dc775ea234676b121af6e5337885833096c15df1531737945eb93c5b558f348"}
Sep 04 23:08:58.233230 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.233166    2577 scope.go:117] "RemoveContainer" containerID="67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"
Sep 04 23:08:58.247801 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.247781    2577 scope.go:117] "RemoveContainer" containerID="67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"
Sep 04 23:08:58.248102 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:08:58.248081    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990\": container with ID starting with 67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990 not found: ID does not exist" containerID="67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"
Sep 04 23:08:58.248164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.248109    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990"} err="failed to get container status \"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990\": rpc error: code = NotFound desc = could not find container \"67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990\": container with ID starting with 67ca96af9af1df81573e3eb677f2b1c25f0a8c41e2dfb69c479d477a42647990 not found: ID does not exist"
Sep 04 23:08:58.259604 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.259585    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:08:58.265004 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.264981    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/auth-test-model-9ec1e5-kserve-84f989d6c-62qxs"]
Sep 04 23:08:58.325289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.325259    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" path="/var/lib/kubelet/pods/7bf4c1e4-11fc-4149-b733-aed3ff42e9f4/volumes"
Sep 04 23:08:58.802645 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:58.802602    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" podUID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" containerName="main" containerID="cri-o://22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4" gracePeriod=45
Sep 04 23:08:59.048998 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.048977    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:08:59.187438 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp8xx\" (UniqueName: \"kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187830 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs\") pod \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\" (UID: \"89b098a1-b16e-47dc-b0de-06e8a2c27c1b\") "
Sep 04 23:08:59.187830 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:59.187943 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:59.187943 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home" (OuterVolumeSpecName: "home") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:59.188028 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187941    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:59.188028 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.187959    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:59.189701 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.189671    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm" (OuterVolumeSpecName: "dshm") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:08:59.189826 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.189726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:08:59.189826 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.189772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx" (OuterVolumeSpecName: "kube-api-access-vp8xx") pod "89b098a1-b16e-47dc-b0de-06e8a2c27c1b" (UID: "89b098a1-b16e-47dc-b0de-06e8a2c27c1b"). InnerVolumeSpecName "kube-api-access-vp8xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:08:59.240138 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.240106    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" containerID="22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4" exitCode=0
Sep 04 23:08:59.240290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.240144    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"
Sep 04 23:08:59.240290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.240161    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" event={"ID":"89b098a1-b16e-47dc-b0de-06e8a2c27c1b","Type":"ContainerDied","Data":"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"}
Sep 04 23:08:59.240290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.240199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x" event={"ID":"89b098a1-b16e-47dc-b0de-06e8a2c27c1b","Type":"ContainerDied","Data":"ded3509da206230bdfdbe0c0779ebd24c85ec178bca96e4296adf64001fa2e38"}
Sep 04 23:08:59.240290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.240217    2577 scope.go:117] "RemoveContainer" containerID="22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"
Sep 04 23:08:59.248961 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.248945    2577 scope.go:117] "RemoveContainer" containerID="22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"
Sep 04 23:08:59.249233 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:08:59.249214    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4\": container with ID starting with 22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4 not found: ID does not exist" containerID="22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"
Sep 04 23:08:59.249280 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.249243    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4"} err="failed to get container status \"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4\": rpc error: code = NotFound desc = could not find container \"22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4\": container with ID starting with 22757eb4efe45cfe20755c14a51498a66c89cd1dac0a9ebb08ec737ae7972ba4 not found: ID does not exist"
Sep 04 23:08:59.260271 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.260248    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:08:59.264725 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.264705    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/auth-test-model-146c0e-kserve-5449d948c5-w8j8x"]
Sep 04 23:08:59.288341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.288322    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:59.288341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.288342    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:59.288479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.288354    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:08:59.288479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:08:59.288365    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8xx\" (UniqueName: \"kubernetes.io/projected/89b098a1-b16e-47dc-b0de-06e8a2c27c1b-kube-api-access-vp8xx\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:00.196194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196159    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:09:00.196648 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196627    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" containerName="main"
Sep 04 23:09:00.196731 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196652    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" containerName="main"
Sep 04 23:09:00.196731 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196665    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" containerName="main"
Sep 04 23:09:00.196731 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196673    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" containerName="main"
Sep 04 23:09:00.196731 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196707    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfffe6e-e232-4cd8-a253-515ce76bb559" containerName="mt-curl-8002-f45970"
Sep 04 23:09:00.196731 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196718    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfffe6e-e232-4cd8-a253-515ce76bb559" containerName="mt-curl-8002-f45970"
Sep 04 23:09:00.197004 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196831    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf4c1e4-11fc-4149-b733-aed3ff42e9f4" containerName="main"
Sep 04 23:09:00.197004 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196844    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" containerName="main"
Sep 04 23:09:00.197004 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.196855    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfffe6e-e232-4cd8-a253-515ce76bb559" containerName="mt-curl-8002-f45970"
Sep 04 23:09:00.201157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.201136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.204498 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.204474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"sub-test-model-9ec1e5-kserve-self-signed-certs\""
Sep 04 23:09:00.204498 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.204488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"kube-root-ca.crt\""
Sep 04 23:09:00.204679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.204488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"default-dockercfg-ql7bt\""
Sep 04 23:09:00.204679 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.204517    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"openshift-service-ca.crt\""
Sep 04 23:09:00.209398 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.209377    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:09:00.297088 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.297281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.297281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.297281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.297281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5z2\" (UniqueName: \"kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.297281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.297257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.325603 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.325571    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b098a1-b16e-47dc-b0de-06e8a2c27c1b" path="/var/lib/kubelet/pods/89b098a1-b16e-47dc-b0de-06e8a2c27c1b/volumes"
Sep 04 23:09:00.397831 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.397968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.397968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.397968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.397968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5z2\" (UniqueName: \"kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.397968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.397929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.398343 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.398314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.398456 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.398352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.398456 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.398432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.400158 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.400140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.400338 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.400321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.406569 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.406546    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5z2\" (UniqueName: \"kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2\") pod \"sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.511909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.511824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:00.781606 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:00.781579    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:09:00.783620 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:09:00.783592    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab17d20_7f0d_4aa8_b07c_2bbe1ec12c7c.slice/crio-e3a6300beb3c2726e741589a7dfd4146dd0ac33fe8bb7e71cdf9a8306c2bb7bb WatchSource:0}: Error finding container e3a6300beb3c2726e741589a7dfd4146dd0ac33fe8bb7e71cdf9a8306c2bb7bb: Status 404 returned error can't find the container with id e3a6300beb3c2726e741589a7dfd4146dd0ac33fe8bb7e71cdf9a8306c2bb7bb
Sep 04 23:09:01.253324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:01.253287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" event={"ID":"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c","Type":"ContainerStarted","Data":"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"}
Sep 04 23:09:01.253674 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:01.253333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" event={"ID":"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c","Type":"ContainerStarted","Data":"e3a6300beb3c2726e741589a7dfd4146dd0ac33fe8bb7e71cdf9a8306c2bb7bb"}
Sep 04 23:09:01.271273 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:01.271229    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" podStartSLOduration=1.271214341 podStartE2EDuration="1.271214341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:09:01.270133083 +0000 UTC m=+1425.441207701" watchObservedRunningTime="2026-09-04 23:09:01.271214341 +0000 UTC m=+1425.442288953"
Sep 04 23:09:03.330281 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.330246    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:09:03.330758 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.330481    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" podUID="52ccf919-b9df-42e1-a859-6ae9dae2645c" containerName="payload-pre-processing" containerID="cri-o://3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9" gracePeriod=30
Sep 04 23:09:03.572871 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.572846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:09:03.725695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.725665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztggc\" (UniqueName: \"kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc\") pod \"52ccf919-b9df-42e1-a859-6ae9dae2645c\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") "
Sep 04 23:09:03.725884 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.725725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume\") pod \"52ccf919-b9df-42e1-a859-6ae9dae2645c\" (UID: \"52ccf919-b9df-42e1-a859-6ae9dae2645c\") "
Sep 04 23:09:03.726127 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.726102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52ccf919-b9df-42e1-a859-6ae9dae2645c" (UID: "52ccf919-b9df-42e1-a859-6ae9dae2645c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:09:03.727689 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.727667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc" (OuterVolumeSpecName: "kube-api-access-ztggc") pod "52ccf919-b9df-42e1-a859-6ae9dae2645c" (UID: "52ccf919-b9df-42e1-a859-6ae9dae2645c"). InnerVolumeSpecName "kube-api-access-ztggc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:09:03.826321 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.826290    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztggc\" (UniqueName: \"kubernetes.io/projected/52ccf919-b9df-42e1-a859-6ae9dae2645c-kube-api-access-ztggc\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:03.826321 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:03.826320    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ccf919-b9df-42e1-a859-6ae9dae2645c-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:04.269789 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.269751    2577 generic.go:356] "Generic (PLEG): container finished" podID="52ccf919-b9df-42e1-a859-6ae9dae2645c" containerID="3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9" exitCode=0
Sep 04 23:09:04.269789 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.269792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" event={"ID":"52ccf919-b9df-42e1-a859-6ae9dae2645c","Type":"ContainerDied","Data":"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"}
Sep 04 23:09:04.270020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.269803    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"
Sep 04 23:09:04.270020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.269836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89" event={"ID":"52ccf919-b9df-42e1-a859-6ae9dae2645c","Type":"ContainerDied","Data":"49e0d865238f2fcc4713edaeef7a0b6a99192bc23fc559dc85282cf124fb3e43"}
Sep 04 23:09:04.270020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.269858    2577 scope.go:117] "RemoveContainer" containerID="3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"
Sep 04 23:09:04.279164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.279145    2577 scope.go:117] "RemoveContainer" containerID="3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"
Sep 04 23:09:04.279448 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:09:04.279424    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9\": container with ID starting with 3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9 not found: ID does not exist" containerID="3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"
Sep 04 23:09:04.279498 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.279457    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9"} err="failed to get container status \"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9\": rpc error: code = NotFound desc = could not find container \"3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9\": container with ID starting with 3493c8296241f85cf91b8169a0fc859ce6160adb06da6a922c5be8fa8b151ff9 not found: ID does not exist"
Sep 04 23:09:04.292621 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.292597    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:09:04.296577 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.296555    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f64c8d3-56578d957d-8gz89"]
Sep 04 23:09:04.325646 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:04.325617    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ccf919-b9df-42e1-a859-6ae9dae2645c" path="/var/lib/kubelet/pods/52ccf919-b9df-42e1-a859-6ae9dae2645c/volumes"
Sep 04 23:09:07.300404 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:07.300373    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:09:10.513012 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:10.512922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:10.513012 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:10.512984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:10.525287 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:10.525258    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:11.317608 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:11.317577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:09:11.695676 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:11.695645    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:09:26.715063 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:26.714992    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" podUID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" containerName="main" containerID="cri-o://02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273" gracePeriod=45
Sep 04 23:09:26.971823 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:26.971765    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:09:27.038031 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.037993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038209 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038074    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038209 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038209 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng527\" (UniqueName: \"kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527\") pod \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\" (UID: \"e5e8e048-a19d-4787-b29e-b00fffa9bd9b\") "
Sep 04 23:09:27.038406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:09:27.038406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:09:27.038406 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home" (OuterVolumeSpecName: "home") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:09:27.038657 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038518    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.038657 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038539    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.038657 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.038552    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.040381 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.040353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm" (OuterVolumeSpecName: "dshm") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:09:27.040491 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.040394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:09:27.040795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.040708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527" (OuterVolumeSpecName: "kube-api-access-ng527") pod "e5e8e048-a19d-4787-b29e-b00fffa9bd9b" (UID: "e5e8e048-a19d-4787-b29e-b00fffa9bd9b"). InnerVolumeSpecName "kube-api-access-ng527". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:09:27.139437 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.139397    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.139437 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.139433    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng527\" (UniqueName: \"kubernetes.io/projected/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-kube-api-access-ng527\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.139437 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.139444    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5e8e048-a19d-4787-b29e-b00fffa9bd9b-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:09:27.376359 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.376320    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" containerID="02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273" exitCode=0
Sep 04 23:09:27.376522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.376364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"
Sep 04 23:09:27.376522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.376405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" event={"ID":"e5e8e048-a19d-4787-b29e-b00fffa9bd9b","Type":"ContainerDied","Data":"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"}
Sep 04 23:09:27.376522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.376454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg" event={"ID":"e5e8e048-a19d-4787-b29e-b00fffa9bd9b","Type":"ContainerDied","Data":"573cb458fd2023cce7406fc30dd091847ff138872d10ca30c6a85f56412895f1"}
Sep 04 23:09:27.376522 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.376472    2577 scope.go:117] "RemoveContainer" containerID="02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"
Sep 04 23:09:27.385582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.385563    2577 scope.go:117] "RemoveContainer" containerID="02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"
Sep 04 23:09:27.385877 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:09:27.385855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273\": container with ID starting with 02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273 not found: ID does not exist" containerID="02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"
Sep 04 23:09:27.385937 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.385885    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273"} err="failed to get container status \"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273\": rpc error: code = NotFound desc = could not find container \"02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273\": container with ID starting with 02c9e4d5fcce3686c010cbc822464a1eb3666fea8cab1f23974b12e9756d1273 not found: ID does not exist"
Sep 04 23:09:27.396579 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.396559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:09:27.400909 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:27.400889    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9f64c8d3/e2e-lifecycle-model-9f64c8d3-kserve-847865f98c-c9rgg"]
Sep 04 23:09:28.325769 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:28.325720    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" path="/var/lib/kubelet/pods/e5e8e048-a19d-4787-b29e-b00fffa9bd9b/volumes"
Sep 04 23:09:38.794278 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794247    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794585    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ccf919-b9df-42e1-a859-6ae9dae2645c" containerName="payload-pre-processing"
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ccf919-b9df-42e1-a859-6ae9dae2645c" containerName="payload-pre-processing"
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794609    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" containerName="main"
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" containerName="main"
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ccf919-b9df-42e1-a859-6ae9dae2645c" containerName="payload-pre-processing"
Sep 04 23:09:38.794766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.794680    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e8e048-a19d-4787-b29e-b00fffa9bd9b" containerName="main"
Sep 04 23:09:38.798816 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.798794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.802174 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.802151    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"kube-root-ca.crt\""
Sep 04 23:09:38.802315 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.802178    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"default-dockercfg-qq6dq\""
Sep 04 23:09:38.802315 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.802190    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"openshift-service-ca.crt\""
Sep 04 23:09:38.802315 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.802310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"sub-test-model-146c0e-kserve-self-signed-certs\""
Sep 04 23:09:38.805657 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.805633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:09:38.943834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.943834 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.944058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.944058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29nd\" (UniqueName: \"kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.944058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:38.944058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:38.943990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.044923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.044986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29nd\" (UniqueName: \"kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.045557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.045456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.047349 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.047329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.047520 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.047504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.053147 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.053120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29nd\" (UniqueName: \"kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd\") pod \"sub-test-model-146c0e-kserve-5557bc9569-jv2sv\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.108508 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.108468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:39.376368 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.376344    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:09:39.378966 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:09:39.378940    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaffe867f_4765_401f_ab67_c6d6d311d9b2.slice/crio-dcc48f67e5c611ca77278d920c15c6e156b5d5a8df3cbd94f7c2b65440b2391a WatchSource:0}: Error finding container dcc48f67e5c611ca77278d920c15c6e156b5d5a8df3cbd94f7c2b65440b2391a: Status 404 returned error can't find the container with id dcc48f67e5c611ca77278d920c15c6e156b5d5a8df3cbd94f7c2b65440b2391a
Sep 04 23:09:39.431840 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:39.431803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" event={"ID":"affe867f-4765-401f-ab67-c6d6d311d9b2","Type":"ContainerStarted","Data":"dcc48f67e5c611ca77278d920c15c6e156b5d5a8df3cbd94f7c2b65440b2391a"}
Sep 04 23:09:40.438172 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:40.438131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" event={"ID":"affe867f-4765-401f-ab67-c6d6d311d9b2","Type":"ContainerStarted","Data":"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"}
Sep 04 23:09:40.455555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:40.455507    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" podStartSLOduration=2.45549398 podStartE2EDuration="2.45549398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:09:40.454148362 +0000 UTC m=+1464.625222973" watchObservedRunningTime="2026-09-04 23:09:40.45549398 +0000 UTC m=+1464.626568611"
Sep 04 23:09:44.798212 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:44.798178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:09:49.108764 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:49.108703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:49.108764 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:49.108773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:49.121053 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:49.121019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:09:49.489460 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:09:49.489366    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:10:02.089780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:02.089730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:10:16.513959 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:16.513925    2577 scope.go:117] "RemoveContainer" containerID="4df502874f4faf8b6fdc7e40e16487edb3829b86acf3d90b433d6769489b4b8d"
Sep 04 23:10:17.787133 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:17.787095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:10:28.973967 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:28.973931    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:10:30.006890 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:30.006852    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:10:31.491450 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:31.491419    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:10:43.993303 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:43.993211    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" podUID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" containerName="main" containerID="cri-o://84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f" gracePeriod=45
Sep 04 23:10:44.241637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.241615    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:10:44.289231 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.289153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:10:44.318932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.318904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.318932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.318937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.319151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.318953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.319151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.318978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.319151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk5z2\" (UniqueName: \"kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.319151 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm\") pod \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\" (UID: \"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c\") "
Sep 04 23:10:44.319363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:44.319363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:44.319363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home" (OuterVolumeSpecName: "home") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:44.319521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319402    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.319521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319420    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.319521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.319435    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.321351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.321315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm" (OuterVolumeSpecName: "dshm") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:44.321471 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.321378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:10:44.322010 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.321989    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2" (OuterVolumeSpecName: "kube-api-access-dk5z2") pod "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" (UID: "fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c"). InnerVolumeSpecName "kube-api-access-dk5z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:10:44.420245 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.420202    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.420245 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.420230    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5z2\" (UniqueName: \"kubernetes.io/projected/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-kube-api-access-dk5z2\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.420245 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.420240    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:44.729007 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.728975    2577 generic.go:356] "Generic (PLEG): container finished" podID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" containerID="84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f" exitCode=0
Sep 04 23:10:44.729165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.729027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"
Sep 04 23:10:44.729165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.729040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" event={"ID":"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c","Type":"ContainerDied","Data":"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"}
Sep 04 23:10:44.729165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.729070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf" event={"ID":"fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c","Type":"ContainerDied","Data":"e3a6300beb3c2726e741589a7dfd4146dd0ac33fe8bb7e71cdf9a8306c2bb7bb"}
Sep 04 23:10:44.729165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.729086    2577 scope.go:117] "RemoveContainer" containerID="84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"
Sep 04 23:10:44.737983 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.737960    2577 scope.go:117] "RemoveContainer" containerID="84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"
Sep 04 23:10:44.738302 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:10:44.738280    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f\": container with ID starting with 84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f not found: ID does not exist" containerID="84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"
Sep 04 23:10:44.738371 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.738313    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f"} err="failed to get container status \"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f\": rpc error: code = NotFound desc = could not find container \"84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f\": container with ID starting with 84b04fbe71e86f7bea288bdf22289ccdbbef7f2a6197c3ea6c7e90532dcd904f not found: ID does not exist"
Sep 04 23:10:44.745259 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.745234    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:10:44.752710 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:44.752689    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/sub-test-model-9ec1e5-kserve-5475bcf9c9-crxqf"]
Sep 04 23:10:45.026188 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.026105    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" podUID="affe867f-4765-401f-ab67-c6d6d311d9b2" containerName="main" containerID="cri-o://85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75" gracePeriod=45
Sep 04 23:10:45.275099 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.275074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:10:45.328768 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.328768 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.328768 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328726    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.328768 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.329103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.329103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.328840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l29nd\" (UniqueName: \"kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd\") pod \"affe867f-4765-401f-ab67-c6d6d311d9b2\" (UID: \"affe867f-4765-401f-ab67-c6d6d311d9b2\") "
Sep 04 23:10:45.329103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.329048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:45.329103 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.329034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:45.329252 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.329109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home" (OuterVolumeSpecName: "home") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:45.330905 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.330880    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm" (OuterVolumeSpecName: "dshm") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:10:45.331017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.330954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:10:45.331017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.330990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd" (OuterVolumeSpecName: "kube-api-access-l29nd") pod "affe867f-4765-401f-ab67-c6d6d311d9b2" (UID: "affe867f-4765-401f-ab67-c6d6d311d9b2"). InnerVolumeSpecName "kube-api-access-l29nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:10:45.430171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430130    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.430171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430162    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.430171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430171    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affe867f-4765-401f-ab67-c6d6d311d9b2-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.430171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430179    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.430171 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430186    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affe867f-4765-401f-ab67-c6d6d311d9b2-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.430461 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.430194    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29nd\" (UniqueName: \"kubernetes.io/projected/affe867f-4765-401f-ab67-c6d6d311d9b2-kube-api-access-l29nd\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:10:45.736197 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.736158    2577 generic.go:356] "Generic (PLEG): container finished" podID="affe867f-4765-401f-ab67-c6d6d311d9b2" containerID="85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75" exitCode=0
Sep 04 23:10:45.736399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.736205    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"
Sep 04 23:10:45.736399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.736210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" event={"ID":"affe867f-4765-401f-ab67-c6d6d311d9b2","Type":"ContainerDied","Data":"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"}
Sep 04 23:10:45.736399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.736318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv" event={"ID":"affe867f-4765-401f-ab67-c6d6d311d9b2","Type":"ContainerDied","Data":"dcc48f67e5c611ca77278d920c15c6e156b5d5a8df3cbd94f7c2b65440b2391a"}
Sep 04 23:10:45.736399 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.736338    2577 scope.go:117] "RemoveContainer" containerID="85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"
Sep 04 23:10:45.744814 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.744797    2577 scope.go:117] "RemoveContainer" containerID="85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"
Sep 04 23:10:45.745057 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:10:45.745035    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75\": container with ID starting with 85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75 not found: ID does not exist" containerID="85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"
Sep 04 23:10:45.745125 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.745067    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75"} err="failed to get container status \"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75\": rpc error: code = NotFound desc = could not find container \"85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75\": container with ID starting with 85bf9f4611248402c33c4c58d4f433412a0da77df52b73690f6a90a958aabb75 not found: ID does not exist"
Sep 04 23:10:45.755700 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.755675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:10:45.758421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:45.758401    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/sub-test-model-146c0e-kserve-5557bc9569-jv2sv"]
Sep 04 23:10:46.331088 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:46.331044    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="affe867f-4765-401f-ab67-c6d6d311d9b2" path="/var/lib/kubelet/pods/affe867f-4765-401f-ab67-c6d6d311d9b2/volumes"
Sep 04 23:10:46.331559 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:46.331541    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" path="/var/lib/kubelet/pods/fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c/volumes"
Sep 04 23:10:47.628008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.627976    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affe867f-4765-401f-ab67-c6d6d311d9b2" containerName="main"
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="affe867f-4765-401f-ab67-c6d6d311d9b2" containerName="main"
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628354    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" containerName="main"
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" containerName="main"
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628419    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="affe867f-4765-401f-ab67-c6d6d311d9b2" containerName="main"
Sep 04 23:10:47.628465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.628427    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab17d20-7f0d-4aa8-b07c-2bbe1ec12c7c" containerName="main"
Sep 04 23:10:47.631229 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.631212    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.633650 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.633631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-17eaa7\""
Sep 04 23:10:47.639604 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.639577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:10:47.650251 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.650227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glh98\" (UniqueName: \"kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.650464 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.650440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.667372 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.667343    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:10:47.670521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.670503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.672780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.672759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-17eaa7-dockercfg-bfqrm\""
Sep 04 23:10:47.680701 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.680682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:10:47.751021 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.750986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glh98\" (UniqueName: \"kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.751186 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.751037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bzp\" (UniqueName: \"kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.751186 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.751057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.751186 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.751149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.751702 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.751686    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.759088 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.759060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glh98\" (UniqueName: \"kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98\") pod \"payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.852237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.852195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.852425 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.852267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bzp\" (UniqueName: \"kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.852944 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.852925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.862086 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.862052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bzp\" (UniqueName: \"kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp\") pod \"payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") " pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:47.942862 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.942784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:47.980092 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:47.980026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:48.213552 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.213526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:10:48.215682 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:10:48.215656    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ab55e4_efc6_45b2_b8f1_e39f7a5bcdcc.slice/crio-b273ee188edc7051b505ca9be11aa0ff6663fb01649902292b93dcd582e0e8b2 WatchSource:0}: Error finding container b273ee188edc7051b505ca9be11aa0ff6663fb01649902292b93dcd582e0e8b2: Status 404 returned error can't find the container with id b273ee188edc7051b505ca9be11aa0ff6663fb01649902292b93dcd582e0e8b2
Sep 04 23:10:48.255036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.254998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:10:48.257900 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:10:48.257843    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ab88f8_13db_4e3d_ac3f_29d7bca6434c.slice/crio-8248b187e15df9d85f904c7a9cb2a1654cea2115bcd510f27cb496757f4cec9d WatchSource:0}: Error finding container 8248b187e15df9d85f904c7a9cb2a1654cea2115bcd510f27cb496757f4cec9d: Status 404 returned error can't find the container with id 8248b187e15df9d85f904c7a9cb2a1654cea2115bcd510f27cb496757f4cec9d
Sep 04 23:10:48.753342 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.753298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" event={"ID":"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc","Type":"ContainerStarted","Data":"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"}
Sep 04 23:10:48.753819 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.753368    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" event={"ID":"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc","Type":"ContainerStarted","Data":"b273ee188edc7051b505ca9be11aa0ff6663fb01649902292b93dcd582e0e8b2"}
Sep 04 23:10:48.753819 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.753396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:48.755093 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.755063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" event={"ID":"80ab88f8-13db-4e3d-ac3f-29d7bca6434c","Type":"ContainerStarted","Data":"40769724239c47a24e98310f4c95515695d249cdee1864f74093f8fe68bc3c96"}
Sep 04 23:10:48.755193 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.755109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" event={"ID":"80ab88f8-13db-4e3d-ac3f-29d7bca6434c","Type":"ContainerStarted","Data":"8248b187e15df9d85f904c7a9cb2a1654cea2115bcd510f27cb496757f4cec9d"}
Sep 04 23:10:48.755193 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.755143    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:48.770357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.770308    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" podStartSLOduration=1.77029003 podStartE2EDuration="1.77029003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:10:48.768974364 +0000 UTC m=+1532.940048977" watchObservedRunningTime="2026-09-04 23:10:48.77029003 +0000 UTC m=+1532.941364642"
Sep 04 23:10:48.785218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:48.785174    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" podStartSLOduration=1.785160221 podStartE2EDuration="1.785160221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:10:48.784476012 +0000 UTC m=+1532.955550628" watchObservedRunningTime="2026-09-04 23:10:48.785160221 +0000 UTC m=+1532.956234839"
Sep 04 23:10:59.761297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:59.761260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:10:59.761806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:59.761347    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:10:59.993924 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:10:59.993888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:11:03.900585 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.900553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:11:03.904010 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.903993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.907559 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.907537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"default-dockercfg-ql7bt\""
Sep 04 23:11:03.907667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.907561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"openshift-service-ca.crt\""
Sep 04 23:11:03.907667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.907540    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"kube-root-ca.crt\""
Sep 04 23:11:03.907667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.907540    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-9ec1e5\"/\"rate-test-model-9ec1e5-kserve-self-signed-certs\""
Sep 04 23:11:03.915780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.915755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:11:03.991420 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscgb\" (UniqueName: \"kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.991420 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.991664 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.991664 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.991664 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:03.991841 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:03.991681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093056 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscgb\" (UniqueName: \"kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093246 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093534 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.093637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.093588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.095381 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.095360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.095686 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.095670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.101081 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.101050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscgb\" (UniqueName: \"kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb\") pod \"rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") " pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.215807 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.215689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:04.483496 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.483472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:11:04.485229 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:11:04.485197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode82cba8d_94bb_4439_b991_de289b9bf66d.slice/crio-b4080a8496947b5a496d17c6074fccdeed135f80d0d2b53b162ba12b4ea7ed91 WatchSource:0}: Error finding container b4080a8496947b5a496d17c6074fccdeed135f80d0d2b53b162ba12b4ea7ed91: Status 404 returned error can't find the container with id b4080a8496947b5a496d17c6074fccdeed135f80d0d2b53b162ba12b4ea7ed91
Sep 04 23:11:04.831799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.831765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" event={"ID":"e82cba8d-94bb-4439-b991-de289b9bf66d","Type":"ContainerStarted","Data":"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"}
Sep 04 23:11:04.831969 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.831813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" event={"ID":"e82cba8d-94bb-4439-b991-de289b9bf66d","Type":"ContainerStarted","Data":"b4080a8496947b5a496d17c6074fccdeed135f80d0d2b53b162ba12b4ea7ed91"}
Sep 04 23:11:04.848670 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:04.848620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" podStartSLOduration=1.8486039920000001 podStartE2EDuration="1.848603992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:11:04.847906176 +0000 UTC m=+1549.018980790" watchObservedRunningTime="2026-09-04 23:11:04.848603992 +0000 UTC m=+1549.019678615"
Sep 04 23:11:14.216446 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:14.216401    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:14.216910 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:14.216469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:14.228818 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:14.228793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:14.889468 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:14.889435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:11:16.893720 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:16.893683    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:11:16.897026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:16.897009    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:16.899948 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:16.899928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9b6db4-dockercfg-q9kg7\""
Sep 04 23:11:16.900026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:16.899956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9b6db4\""
Sep 04 23:11:16.907965 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:16.907940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:11:17.006820 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.006783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpdks\" (UniqueName: \"kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.006979 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.006864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.108116 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.108084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpdks\" (UniqueName: \"kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.108267 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.108147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.108803 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.108784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.119518 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.119484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpdks\" (UniqueName: \"kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks\") pod \"payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.208135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.208044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.475237 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.475212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:11:17.477199 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:11:17.477176    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8200d8_c1f6_4e00_be3c_6b6f4d934bec.slice/crio-a5ab9dd34e6b0a7835ada7cab601715c5a568b60d494ecec6b1d2dd84b17804f WatchSource:0}: Error finding container a5ab9dd34e6b0a7835ada7cab601715c5a568b60d494ecec6b1d2dd84b17804f: Status 404 returned error can't find the container with id a5ab9dd34e6b0a7835ada7cab601715c5a568b60d494ecec6b1d2dd84b17804f
Sep 04 23:11:17.893280 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.893245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" event={"ID":"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec","Type":"ContainerStarted","Data":"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"}
Sep 04 23:11:17.893456 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.893307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" event={"ID":"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec","Type":"ContainerStarted","Data":"a5ab9dd34e6b0a7835ada7cab601715c5a568b60d494ecec6b1d2dd84b17804f"}
Sep 04 23:11:17.893456 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.893347    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:17.909846 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:17.909799    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" podStartSLOduration=1.9097853489999999 podStartE2EDuration="1.909785349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:11:17.908168641 +0000 UTC m=+1562.079243253" watchObservedRunningTime="2026-09-04 23:11:17.909785349 +0000 UTC m=+1562.080859955"
Sep 04 23:11:18.391510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:18.391476    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:11:28.900651 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:28.900619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:11:32.106810 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:32.106773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:11:45.981932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.981895    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:11:45.986545 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.986523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:45.990002 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.989981    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"openshift-service-ca.crt\""
Sep 04 23:11:45.990234 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.990204    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"default-dockercfg-qq6dq\""
Sep 04 23:11:45.990329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.990207    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"rate-test-model-146c0e-kserve-self-signed-certs\""
Sep 04 23:11:45.990329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.990211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-146c0e\"/\"kube-root-ca.crt\""
Sep 04 23:11:45.995812 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:45.995789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:11:46.036369 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.036554 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.036554 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.036554 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8lh\" (UniqueName: \"kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.036682 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.036682 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.036588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8lh\" (UniqueName: \"kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137929 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.137929 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.138029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.138029 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.137995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.138226 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.138208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.140028 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.140009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.140191 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.140176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.145723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.145703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8lh\" (UniqueName: \"kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh\") pod \"rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") " pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.296673 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.296577    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:46.565931 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:46.565887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:11:46.568242 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:11:46.568215    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd38078d_f3f3_4b13_9072_a5ff0f304acf.slice/crio-883b76851a7ee6d6efb17d094b7c04476029f86d62c5dbe3e323612b386d3b2d WatchSource:0}: Error finding container 883b76851a7ee6d6efb17d094b7c04476029f86d62c5dbe3e323612b386d3b2d: Status 404 returned error can't find the container with id 883b76851a7ee6d6efb17d094b7c04476029f86d62c5dbe3e323612b386d3b2d
Sep 04 23:11:47.024716 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:47.024675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" event={"ID":"bd38078d-f3f3-4b13-9072-a5ff0f304acf","Type":"ContainerStarted","Data":"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"}
Sep 04 23:11:47.025106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:47.024723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" event={"ID":"bd38078d-f3f3-4b13-9072-a5ff0f304acf","Type":"ContainerStarted","Data":"883b76851a7ee6d6efb17d094b7c04476029f86d62c5dbe3e323612b386d3b2d"}
Sep 04 23:11:47.042461 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:47.042409    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" podStartSLOduration=2.042394196 podStartE2EDuration="2.042394196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:11:47.042208064 +0000 UTC m=+1591.213282677" watchObservedRunningTime="2026-09-04 23:11:47.042394196 +0000 UTC m=+1591.213468818"
Sep 04 23:11:52.998466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:52.998422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:11:56.297294 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:56.297249    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:56.297695 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:56.297309    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:56.309912 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:56.309886    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:57.082366 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:57.082332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:11:58.967363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:58.967327    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:11:58.967872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:58.967560    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" podUID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" containerName="payload-processing" containerID="cri-o://40769724239c47a24e98310f4c95515695d249cdee1864f74093f8fe68bc3c96" gracePeriod=30
Sep 04 23:11:58.972908 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:58.972876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:11:58.973161 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:58.973115    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" podUID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" containerName="payload-pre-processing" containerID="cri-o://886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6" gracePeriod=30
Sep 04 23:11:59.083002 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.082973    2577 generic.go:356] "Generic (PLEG): container finished" podID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" containerID="40769724239c47a24e98310f4c95515695d249cdee1864f74093f8fe68bc3c96" exitCode=0
Sep 04 23:11:59.083117 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.083049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" event={"ID":"80ab88f8-13db-4e3d-ac3f-29d7bca6434c","Type":"ContainerDied","Data":"40769724239c47a24e98310f4c95515695d249cdee1864f74093f8fe68bc3c96"}
Sep 04 23:11:59.235602 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.235580    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:11:59.237910 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.237889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume\") pod \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") "
Sep 04 23:11:59.238009 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.237921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bzp\" (UniqueName: \"kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp\") pod \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\" (UID: \"80ab88f8-13db-4e3d-ac3f-29d7bca6434c\") "
Sep 04 23:11:59.238218 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.238194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80ab88f8-13db-4e3d-ac3f-29d7bca6434c" (UID: "80ab88f8-13db-4e3d-ac3f-29d7bca6434c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:11:59.238571 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.238552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:11:59.239840 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.239820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp" (OuterVolumeSpecName: "kube-api-access-46bzp") pod "80ab88f8-13db-4e3d-ac3f-29d7bca6434c" (UID: "80ab88f8-13db-4e3d-ac3f-29d7bca6434c"). InnerVolumeSpecName "kube-api-access-46bzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:11:59.339069 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.339019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glh98\" (UniqueName: \"kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98\") pod \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") "
Sep 04 23:11:59.339261 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.339100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume\") pod \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\" (UID: \"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc\") "
Sep 04 23:11:59.339309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.339276    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:11:59.339309 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.339286    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bzp\" (UniqueName: \"kubernetes.io/projected/80ab88f8-13db-4e3d-ac3f-29d7bca6434c-kube-api-access-46bzp\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:11:59.339496 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.339470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" (UID: "85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:11:59.341159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.341141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98" (OuterVolumeSpecName: "kube-api-access-glh98") pod "85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" (UID: "85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc"). InnerVolumeSpecName "kube-api-access-glh98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:11:59.440143 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.440112    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glh98\" (UniqueName: \"kubernetes.io/projected/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-kube-api-access-glh98\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:11:59.440143 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:11:59.440137    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:00.090013 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.089982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"
Sep 04 23:12:00.090013 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.089990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45" event={"ID":"80ab88f8-13db-4e3d-ac3f-29d7bca6434c","Type":"ContainerDied","Data":"8248b187e15df9d85f904c7a9cb2a1654cea2115bcd510f27cb496757f4cec9d"}
Sep 04 23:12:00.090653 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.090048    2577 scope.go:117] "RemoveContainer" containerID="40769724239c47a24e98310f4c95515695d249cdee1864f74093f8fe68bc3c96"
Sep 04 23:12:00.091440 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.091420    2577 generic.go:356] "Generic (PLEG): container finished" podID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" containerID="886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6" exitCode=0
Sep 04 23:12:00.091512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.091456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" event={"ID":"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc","Type":"ContainerDied","Data":"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"}
Sep 04 23:12:00.091512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.091472    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"
Sep 04 23:12:00.091512 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.091484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v" event={"ID":"85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc","Type":"ContainerDied","Data":"b273ee188edc7051b505ca9be11aa0ff6663fb01649902292b93dcd582e0e8b2"}
Sep 04 23:12:00.107108 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.107089    2577 scope.go:117] "RemoveContainer" containerID="886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"
Sep 04 23:12:00.114865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.114842    2577 scope.go:117] "RemoveContainer" containerID="886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"
Sep 04 23:12:00.115150 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:00.115129    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6\": container with ID starting with 886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6 not found: ID does not exist" containerID="886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"
Sep 04 23:12:00.115215 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.115161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6"} err="failed to get container status \"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6\": rpc error: code = NotFound desc = could not find container \"886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6\": container with ID starting with 886081ad2b37b4e96c255afe1a1beecfbe06fbf0f33d7211e618972f52d26cc6 not found: ID does not exist"
Sep 04 23:12:00.116069 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.116039    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:12:00.118071 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.118042    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17eaa7-5dd57b7d7f-tlx45"]
Sep 04 23:12:00.132599 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.132578    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:12:00.137038 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.137018    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-17eaa7-df8fdd55d-7js5v"]
Sep 04 23:12:00.325748 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.325659    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" path="/var/lib/kubelet/pods/80ab88f8-13db-4e3d-ac3f-29d7bca6434c/volumes"
Sep 04 23:12:00.326065 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:00.326046    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" path="/var/lib/kubelet/pods/85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc/volumes"
Sep 04 23:12:05.865242 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865203    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"]
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865537    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" containerName="payload-processing"
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865547    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" containerName="payload-processing"
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865557    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" containerName="payload-pre-processing"
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865564    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" containerName="payload-pre-processing"
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865620    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ab88f8-13db-4e3d-ac3f-29d7bca6434c" containerName="payload-processing"
Sep 04 23:12:05.865632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.865631    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ab55e4-efc6-45b2-b8f1-e39f7a5bcdcc" containerName="payload-pre-processing"
Sep 04 23:12:05.869993 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.869974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:05.872548 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.872520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ncjwg\""
Sep 04 23:12:05.872687 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.872571    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:12:05.885239 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.885208    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"]
Sep 04 23:12:05.900026 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:05.899984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc\" (UID: \"f65f4322-2c06-4e92-8108-4c3ee088d563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:06.000906 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:06.000862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc\" (UID: \"f65f4322-2c06-4e92-8108-4c3ee088d563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:06.001511 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:06.001489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc\" (UID: \"f65f4322-2c06-4e92-8108-4c3ee088d563\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:06.187405 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:06.187313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:06.454305 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:06.454201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"]
Sep 04 23:12:06.456353 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:12:06.456330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65f4322_2c06_4e92_8108_4c3ee088d563.slice/crio-d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04 WatchSource:0}: Error finding container d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04: Status 404 returned error can't find the container with id d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04
Sep 04 23:12:07.128453 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:07.128411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc" event={"ID":"f65f4322-2c06-4e92-8108-4c3ee088d563","Type":"ContainerStarted","Data":"53863b4fa0e77f46011f59c7235dca9675ac25dc28391f8941a0d93b9e59db04"}
Sep 04 23:12:07.128935 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:07.128475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc" event={"ID":"f65f4322-2c06-4e92-8108-4c3ee088d563","Type":"ContainerStarted","Data":"d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04"}
Sep 04 23:12:07.145799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:07.145727    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc" podStartSLOduration=2.145709199 podStartE2EDuration="2.145709199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:12:07.14415331 +0000 UTC m=+1611.315227922" watchObservedRunningTime="2026-09-04 23:12:07.145709199 +0000 UTC m=+1611.316783888"
Sep 04 23:12:08.134749 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:08.134659    2577 generic.go:356] "Generic (PLEG): container finished" podID="f65f4322-2c06-4e92-8108-4c3ee088d563" containerID="53863b4fa0e77f46011f59c7235dca9675ac25dc28391f8941a0d93b9e59db04" exitCode=0
Sep 04 23:12:08.135108 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:08.134723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc" event={"ID":"f65f4322-2c06-4e92-8108-4c3ee088d563","Type":"ContainerDied","Data":"53863b4fa0e77f46011f59c7235dca9675ac25dc28391f8941a0d93b9e59db04"}
Sep 04 23:12:08.691286 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:08.691246    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:12:09.269066 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:09.269045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:09.328074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:09.328042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca\") pod \"f65f4322-2c06-4e92-8108-4c3ee088d563\" (UID: \"f65f4322-2c06-4e92-8108-4c3ee088d563\") "
Sep 04 23:12:09.328423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:09.328404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f65f4322-2c06-4e92-8108-4c3ee088d563" (UID: "f65f4322-2c06-4e92-8108-4c3ee088d563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:12:09.429194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:09.429095    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65f4322-2c06-4e92-8108-4c3ee088d563-maas-api-service-ca\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:10.145809 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:10.145771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc" event={"ID":"f65f4322-2c06-4e92-8108-4c3ee088d563","Type":"ContainerDied","Data":"d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04"}
Sep 04 23:12:10.145809 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:10.145792    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"
Sep 04 23:12:10.146020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:10.145842    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d14217940eb371b3839c43d1ecaa63825139bb95375864e00b640b9b95e4bf04"
Sep 04 23:12:21.508620 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.508583    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:12:21.509102 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.508843    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" podUID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" containerName="payload-pre-processing" containerID="cri-o://499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245" gracePeriod=30
Sep 04 23:12:21.662221 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.662177    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"]
Sep 04 23:12:21.668649 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.668623    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9b6db4-x9ptc"]
Sep 04 23:12:21.756263 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.756182    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:12:21.842092 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.842047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume\") pod \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") "
Sep 04 23:12:21.842394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.842374    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpdks\" (UniqueName: \"kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks\") pod \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\" (UID: \"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec\") "
Sep 04 23:12:21.843430 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.843384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" (UID: "ee8200d8-c1f6-4e00-be3c-6b6f4d934bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:12:21.849478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.849446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks" (OuterVolumeSpecName: "kube-api-access-mpdks") pod "ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" (UID: "ee8200d8-c1f6-4e00-be3c-6b6f4d934bec"). InnerVolumeSpecName "kube-api-access-mpdks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:12:21.943330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.943277    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:21.943330 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:21.943324    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpdks\" (UniqueName: \"kubernetes.io/projected/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec-kube-api-access-mpdks\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:22.200586 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.200499    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" containerID="499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245" exitCode=0
Sep 04 23:12:22.200586 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.200560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" event={"ID":"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec","Type":"ContainerDied","Data":"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"}
Sep 04 23:12:22.200586 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.200573    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"
Sep 04 23:12:22.200863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.200626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9" event={"ID":"ee8200d8-c1f6-4e00-be3c-6b6f4d934bec","Type":"ContainerDied","Data":"a5ab9dd34e6b0a7835ada7cab601715c5a568b60d494ecec6b1d2dd84b17804f"}
Sep 04 23:12:22.200863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.200647    2577 scope.go:117] "RemoveContainer" containerID="499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"
Sep 04 23:12:22.210047 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.210027    2577 scope.go:117] "RemoveContainer" containerID="499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"
Sep 04 23:12:22.210332 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:22.210313    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245\": container with ID starting with 499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245 not found: ID does not exist" containerID="499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"
Sep 04 23:12:22.210372 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.210342    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245"} err="failed to get container status \"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245\": rpc error: code = NotFound desc = could not find container \"499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245\": container with ID starting with 499046ee3b08176390ec7a8bbf5398b2e663049c1438671c83f3fd5a0ca58245 not found: ID does not exist"
Sep 04 23:12:22.223603 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.223582    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:12:22.227378 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.227357    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9b6db4-5f564f7f7d-g96g9"]
Sep 04 23:12:22.325398 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.325356    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" path="/var/lib/kubelet/pods/ee8200d8-c1f6-4e00-be3c-6b6f4d934bec/volumes"
Sep 04 23:12:22.325795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:22.325774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65f4322-2c06-4e92-8108-4c3ee088d563" path="/var/lib/kubelet/pods/f65f4322-2c06-4e92-8108-4c3ee088d563/volumes"
Sep 04 23:12:29.592820 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:29.592780    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:12:33.496506 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:33.496458    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:12:34.246900 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:34.246863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:12:39.519574 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519537    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:12:39.519986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65f4322-2c06-4e92-8108-4c3ee088d563" containerName="revoke-keys"
Sep 04 23:12:39.519986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519918    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65f4322-2c06-4e92-8108-4c3ee088d563" containerName="revoke-keys"
Sep 04 23:12:39.519986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519933    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" containerName="payload-pre-processing"
Sep 04 23:12:39.519986 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519938    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" containerName="payload-pre-processing"
Sep 04 23:12:39.520135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.519992    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65f4322-2c06-4e92-8108-4c3ee088d563" containerName="revoke-keys"
Sep 04 23:12:39.520135 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.520001    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8200d8-c1f6-4e00-be3c-6b6f4d934bec" containerName="payload-pre-processing"
Sep 04 23:12:39.523945 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.523927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.526232 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.526203    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3543cf\""
Sep 04 23:12:39.526367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.526236    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3543cf-dockercfg-67nzm\""
Sep 04 23:12:39.530483 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.530459    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:12:39.602566 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.602530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.602726 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.602573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntm4\" (UniqueName: \"kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.685780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.685747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:12:39.685981 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.685956    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" podUID="77bb34c3-68fe-45e4-bd0c-816b329d87af" containerName="payload-processing" containerID="cri-o://639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957" gracePeriod=30
Sep 04 23:12:39.703148 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.703119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.703243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.703168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntm4\" (UniqueName: \"kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.704033 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.704008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.714035 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.714012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntm4\" (UniqueName: \"kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4\") pod \"payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.834643 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.834610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:39.942726 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:39.942702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:12:40.005346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.005312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume\") pod \"77bb34c3-68fe-45e4-bd0c-816b329d87af\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") "
Sep 04 23:12:40.005513 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.005364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7fvq\" (UniqueName: \"kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq\") pod \"77bb34c3-68fe-45e4-bd0c-816b329d87af\" (UID: \"77bb34c3-68fe-45e4-bd0c-816b329d87af\") "
Sep 04 23:12:40.005800 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.005765    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77bb34c3-68fe-45e4-bd0c-816b329d87af" (UID: "77bb34c3-68fe-45e4-bd0c-816b329d87af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:12:40.007503 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.007484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq" (OuterVolumeSpecName: "kube-api-access-p7fvq") pod "77bb34c3-68fe-45e4-bd0c-816b329d87af" (UID: "77bb34c3-68fe-45e4-bd0c-816b329d87af"). InnerVolumeSpecName "kube-api-access-p7fvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:12:40.106492 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.106461    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7fvq\" (UniqueName: \"kubernetes.io/projected/77bb34c3-68fe-45e4-bd0c-816b329d87af-kube-api-access-p7fvq\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:40.106492 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.106488    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77bb34c3-68fe-45e4-bd0c-816b329d87af-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:40.129685 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.129652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:12:40.132515 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:12:40.132484    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd3cd9a_c9d0_42aa_b3ce_8562752f9cc5.slice/crio-1168a05f2829fd27ba720ff8c2158ba746666213107234588350c3cb1991059e WatchSource:0}: Error finding container 1168a05f2829fd27ba720ff8c2158ba746666213107234588350c3cb1991059e: Status 404 returned error can't find the container with id 1168a05f2829fd27ba720ff8c2158ba746666213107234588350c3cb1991059e
Sep 04 23:12:40.284145 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.284108    2577 generic.go:356] "Generic (PLEG): container finished" podID="77bb34c3-68fe-45e4-bd0c-816b329d87af" containerID="639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957" exitCode=0
Sep 04 23:12:40.284145 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.284149    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"
Sep 04 23:12:40.284373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.284196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" event={"ID":"77bb34c3-68fe-45e4-bd0c-816b329d87af","Type":"ContainerDied","Data":"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"}
Sep 04 23:12:40.284373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.284251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb" event={"ID":"77bb34c3-68fe-45e4-bd0c-816b329d87af","Type":"ContainerDied","Data":"1a335503e0afe2888f3339dde356694524d5155dcf20df2f70b4ba299db433f6"}
Sep 04 23:12:40.284373 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.284283    2577 scope.go:117] "RemoveContainer" containerID="639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"
Sep 04 23:12:40.286037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.286014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" event={"ID":"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5","Type":"ContainerStarted","Data":"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"}
Sep 04 23:12:40.286153 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.286074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" event={"ID":"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5","Type":"ContainerStarted","Data":"1168a05f2829fd27ba720ff8c2158ba746666213107234588350c3cb1991059e"}
Sep 04 23:12:40.286153 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.286105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:40.298224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.298206    2577 scope.go:117] "RemoveContainer" containerID="639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"
Sep 04 23:12:40.298519 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:40.298494    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957\": container with ID starting with 639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957 not found: ID does not exist" containerID="639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"
Sep 04 23:12:40.298617 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.298522    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957"} err="failed to get container status \"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957\": rpc error: code = NotFound desc = could not find container \"639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957\": container with ID starting with 639eaf05c7a7e1299b13b79f7fc50c8874f24fcb7ae8dac6e8c390346afb0957 not found: ID does not exist"
Sep 04 23:12:40.308995 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.308950    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" podStartSLOduration=1.308936002 podStartE2EDuration="1.308936002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:12:40.306976509 +0000 UTC m=+1644.478051124" watchObservedRunningTime="2026-09-04 23:12:40.308936002 +0000 UTC m=+1644.480010614"
Sep 04 23:12:40.325430 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.325404    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:12:40.325430 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:40.325438    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ed43d3-868c9c7d75-5hxcb"]
Sep 04 23:12:42.325491 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:42.325455    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77bb34c3-68fe-45e4-bd0c-816b329d87af" path="/var/lib/kubelet/pods/77bb34c3-68fe-45e4-bd0c-816b329d87af/volumes"
Sep 04 23:12:44.599123 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:44.599088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:12:47.312047 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.312010    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"]
Sep 04 23:12:47.312462 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.312367    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77bb34c3-68fe-45e4-bd0c-816b329d87af" containerName="payload-processing"
Sep 04 23:12:47.312462 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.312379    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77bb34c3-68fe-45e4-bd0c-816b329d87af" containerName="payload-processing"
Sep 04 23:12:47.312462 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.312431    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77bb34c3-68fe-45e4-bd0c-816b329d87af" containerName="payload-processing"
Sep 04 23:12:47.315325 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.315306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:47.317746 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.317719    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ncjwg\""
Sep 04 23:12:47.317857 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.317719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 23:12:47.323665 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.323641    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"]
Sep 04 23:12:47.358832 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.358794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7\" (UID: \"f2455a1c-6bdf-4458-894c-3ead12793a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:47.459938 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.459901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7\" (UID: \"f2455a1c-6bdf-4458-894c-3ead12793a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:47.460582 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.460560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7\" (UID: \"f2455a1c-6bdf-4458-894c-3ead12793a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:47.625781 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.625730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:47.902003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:47.901964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"]
Sep 04 23:12:47.907768 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:12:47.907722    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2455a1c_6bdf_4458_894c_3ead12793a74.slice/crio-ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b WatchSource:0}: Error finding container ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b: Status 404 returned error can't find the container with id ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b
Sep 04 23:12:48.325015 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.324978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7" event={"ID":"f2455a1c-6bdf-4458-894c-3ead12793a74","Type":"ContainerStarted","Data":"857ce4c06c5b769097df85bbd4514c06cac82af725bcf1f706558a7786d06b6f"}
Sep 04 23:12:48.325374 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.325028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7" event={"ID":"f2455a1c-6bdf-4458-894c-3ead12793a74","Type":"ContainerStarted","Data":"ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b"}
Sep 04 23:12:48.340180 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.340137    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7" podStartSLOduration=1.340121871 podStartE2EDuration="1.340121871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:12:48.338279545 +0000 UTC m=+1652.509354156" watchObservedRunningTime="2026-09-04 23:12:48.340121871 +0000 UTC m=+1652.511196482"
Sep 04 23:12:48.515975 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.515886    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" podUID="e82cba8d-94bb-4439-b991-de289b9bf66d" containerName="main" containerID="cri-o://f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34" gracePeriod=45
Sep 04 23:12:48.768162 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.768099    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:12:48.769141 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kscgb\" (UniqueName: \"kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769312 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769312 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769312 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache\") pod \"e82cba8d-94bb-4439-b991-de289b9bf66d\" (UID: \"e82cba8d-94bb-4439-b991-de289b9bf66d\") "
Sep 04 23:12:48.769454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:48.769454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home" (OuterVolumeSpecName: "home") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:48.769572 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769505    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:48.769572 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769521    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:48.769675 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.769638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:48.771418 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.771390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm" (OuterVolumeSpecName: "dshm") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:48.771523 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.771436    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb" (OuterVolumeSpecName: "kube-api-access-kscgb") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "kube-api-access-kscgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:12:48.771523 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.771479    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e82cba8d-94bb-4439-b991-de289b9bf66d" (UID: "e82cba8d-94bb-4439-b991-de289b9bf66d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:12:48.870017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.869970    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:48.870017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.870013    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e82cba8d-94bb-4439-b991-de289b9bf66d-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:48.870017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.870025    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscgb\" (UniqueName: \"kubernetes.io/projected/e82cba8d-94bb-4439-b991-de289b9bf66d-kube-api-access-kscgb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:48.870017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:48.870034    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e82cba8d-94bb-4439-b991-de289b9bf66d-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.266157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.266100    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" podUID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" containerName="main" containerID="cri-o://dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0" gracePeriod=45
Sep 04 23:12:49.331609 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.331577    2577 generic.go:356] "Generic (PLEG): container finished" podID="e82cba8d-94bb-4439-b991-de289b9bf66d" containerID="f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34" exitCode=0
Sep 04 23:12:49.332091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.331611    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"
Sep 04 23:12:49.332091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.331612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" event={"ID":"e82cba8d-94bb-4439-b991-de289b9bf66d","Type":"ContainerDied","Data":"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"}
Sep 04 23:12:49.332091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.331657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52" event={"ID":"e82cba8d-94bb-4439-b991-de289b9bf66d","Type":"ContainerDied","Data":"b4080a8496947b5a496d17c6074fccdeed135f80d0d2b53b162ba12b4ea7ed91"}
Sep 04 23:12:49.332091 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.331690    2577 scope.go:117] "RemoveContainer" containerID="f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"
Sep 04 23:12:49.336438 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.336414    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2455a1c-6bdf-4458-894c-3ead12793a74" containerID="857ce4c06c5b769097df85bbd4514c06cac82af725bcf1f706558a7786d06b6f" exitCode=0
Sep 04 23:12:49.336562 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.336450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7" event={"ID":"f2455a1c-6bdf-4458-894c-3ead12793a74","Type":"ContainerDied","Data":"857ce4c06c5b769097df85bbd4514c06cac82af725bcf1f706558a7786d06b6f"}
Sep 04 23:12:49.365114 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.365075    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:12:49.366876 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.366850    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-9ec1e5/rate-test-model-9ec1e5-kserve-85fc9d759f-5nh52"]
Sep 04 23:12:49.376879 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.376854    2577 scope.go:117] "RemoveContainer" containerID="f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"
Sep 04 23:12:49.377219 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:49.377196    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34\": container with ID starting with f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34 not found: ID does not exist" containerID="f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"
Sep 04 23:12:49.377289 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.377229    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34"} err="failed to get container status \"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34\": rpc error: code = NotFound desc = could not find container \"f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34\": container with ID starting with f3bfd3949cbb0a2454f43b1b9f218a598e81a83230ca4ad3491492020bec1b34 not found: ID does not exist"
Sep 04 23:12:49.498219 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.498195    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:12:49.575036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.574948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.574998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575036 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj8lh\" (UniqueName: \"kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh\") pod \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\" (UID: \"bd38078d-f3f3-4b13-9072-a5ff0f304acf\") "
Sep 04 23:12:49.575367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:49.575367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home" (OuterVolumeSpecName: "home") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:49.575367 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575300    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:49.575594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575401    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.575594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575419    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.575594 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.575436    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.577169 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.577149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm" (OuterVolumeSpecName: "dshm") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:12:49.577236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.577198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:12:49.577236 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.577201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh" (OuterVolumeSpecName: "kube-api-access-qj8lh") pod "bd38078d-f3f3-4b13-9072-a5ff0f304acf" (UID: "bd38078d-f3f3-4b13-9072-a5ff0f304acf"). InnerVolumeSpecName "kube-api-access-qj8lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:12:49.675955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.675903    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd38078d-f3f3-4b13-9072-a5ff0f304acf-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.675955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.675949    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8lh\" (UniqueName: \"kubernetes.io/projected/bd38078d-f3f3-4b13-9072-a5ff0f304acf-kube-api-access-qj8lh\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:49.675955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:49.675960    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd38078d-f3f3-4b13-9072-a5ff0f304acf-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:50.325007 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.324973    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82cba8d-94bb-4439-b991-de289b9bf66d" path="/var/lib/kubelet/pods/e82cba8d-94bb-4439-b991-de289b9bf66d/volumes"
Sep 04 23:12:50.343568 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.343536    2577 generic.go:356] "Generic (PLEG): container finished" podID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" containerID="dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0" exitCode=0
Sep 04 23:12:50.343937 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.343580    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"
Sep 04 23:12:50.343937 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.343621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" event={"ID":"bd38078d-f3f3-4b13-9072-a5ff0f304acf","Type":"ContainerDied","Data":"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"}
Sep 04 23:12:50.343937 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.343664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v" event={"ID":"bd38078d-f3f3-4b13-9072-a5ff0f304acf","Type":"ContainerDied","Data":"883b76851a7ee6d6efb17d094b7c04476029f86d62c5dbe3e323612b386d3b2d"}
Sep 04 23:12:50.343937 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.343688    2577 scope.go:117] "RemoveContainer" containerID="dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"
Sep 04 23:12:50.352317 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.352298    2577 scope.go:117] "RemoveContainer" containerID="dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"
Sep 04 23:12:50.352616 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:50.352601    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0\": container with ID starting with dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0 not found: ID does not exist" containerID="dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"
Sep 04 23:12:50.352660 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.352624    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0"} err="failed to get container status \"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0\": rpc error: code = NotFound desc = could not find container \"dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0\": container with ID starting with dde2901996a3103283a7077766d3102a5031808e910fa64a654d2f66e75d5bc0 not found: ID does not exist"
Sep 04 23:12:50.361188 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.361163    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:12:50.365622 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.365596    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-146c0e/rate-test-model-146c0e-kserve-7b4cf44cff-6vr4v"]
Sep 04 23:12:50.478094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.478072    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:50.482717 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.482696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca\") pod \"f2455a1c-6bdf-4458-894c-3ead12793a74\" (UID: \"f2455a1c-6bdf-4458-894c-3ead12793a74\") "
Sep 04 23:12:50.483011 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.482988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2455a1c-6bdf-4458-894c-3ead12793a74" (UID: "f2455a1c-6bdf-4458-894c-3ead12793a74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:12:50.584130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:50.584103    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2455a1c-6bdf-4458-894c-3ead12793a74-maas-api-service-ca\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:51.293293 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:51.293265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:12:51.351468 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:51.351434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7" event={"ID":"f2455a1c-6bdf-4458-894c-3ead12793a74","Type":"ContainerDied","Data":"ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b"}
Sep 04 23:12:51.351468 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:51.351466    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"
Sep 04 23:12:51.351995 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:51.351481    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae02c2a4431547a7278205c532132c95ccff35c15c57edabd80f54bbc9da374b"
Sep 04 23:12:52.325831 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:52.325797    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" path="/var/lib/kubelet/pods/bd38078d-f3f3-4b13-9072-a5ff0f304acf/volumes"
Sep 04 23:12:57.844643 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:57.844600    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:12:57.845115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:57.844876    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" podUID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" containerName="payload-pre-processing" containerID="cri-o://df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de" gracePeriod=30
Sep 04 23:12:58.005468 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.005437    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"]
Sep 04 23:12:58.009272 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.009247    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-217ad6-7zdl7"]
Sep 04 23:12:58.095776 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.095704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:12:58.146256 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.146222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g8t8\" (UniqueName: \"kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8\") pod \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") "
Sep 04 23:12:58.146440 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.146309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume\") pod \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\" (UID: \"036d6b92-1f51-4bd4-86c6-8c20c9601e38\") "
Sep 04 23:12:58.146696 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.146670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "036d6b92-1f51-4bd4-86c6-8c20c9601e38" (UID: "036d6b92-1f51-4bd4-86c6-8c20c9601e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:12:58.148379 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.148358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8" (OuterVolumeSpecName: "kube-api-access-8g8t8") pod "036d6b92-1f51-4bd4-86c6-8c20c9601e38" (UID: "036d6b92-1f51-4bd4-86c6-8c20c9601e38"). InnerVolumeSpecName "kube-api-access-8g8t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:12:58.247363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.247311    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036d6b92-1f51-4bd4-86c6-8c20c9601e38-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:58.247363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.247358    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8t8\" (UniqueName: \"kubernetes.io/projected/036d6b92-1f51-4bd4-86c6-8c20c9601e38-kube-api-access-8g8t8\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:12:58.324801 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.324765    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2455a1c-6bdf-4458-894c-3ead12793a74" path="/var/lib/kubelet/pods/f2455a1c-6bdf-4458-894c-3ead12793a74/volumes"
Sep 04 23:12:58.382976 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.382897    2577 generic.go:356] "Generic (PLEG): container finished" podID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" containerID="df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de" exitCode=0
Sep 04 23:12:58.382976 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.382938    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"
Sep 04 23:12:58.383200 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.382992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" event={"ID":"036d6b92-1f51-4bd4-86c6-8c20c9601e38","Type":"ContainerDied","Data":"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"}
Sep 04 23:12:58.383200 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.383047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286" event={"ID":"036d6b92-1f51-4bd4-86c6-8c20c9601e38","Type":"ContainerDied","Data":"9c37cf20963a98a043af6c2db10eaf8c23c786d57bc65a6299df74a53d9bba94"}
Sep 04 23:12:58.383200 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.383075    2577 scope.go:117] "RemoveContainer" containerID="df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"
Sep 04 23:12:58.395607 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.391957    2577 scope.go:117] "RemoveContainer" containerID="df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"
Sep 04 23:12:58.395959 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:12:58.395935    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de\": container with ID starting with df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de not found: ID does not exist" containerID="df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"
Sep 04 23:12:58.396062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.395966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de"} err="failed to get container status \"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de\": rpc error: code = NotFound desc = could not find container \"df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de\": container with ID starting with df839826fb425fe8553ad0ecd60f9b4b601eed5b0bab17308aa91a3f0de836de not found: ID does not exist"
Sep 04 23:12:58.400402 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.400382    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:12:58.404323 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:12:58.404302    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-217ad6-86455cd58-cv286"]
Sep 04 23:13:00.324947 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:00.324914    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" path="/var/lib/kubelet/pods/036d6b92-1f51-4bd4-86c6-8c20c9601e38/volumes"
Sep 04 23:13:00.497048 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:00.497013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:13:03.525564 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525523    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525868    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82cba8d-94bb-4439-b991-de289b9bf66d" containerName="main"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525881    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82cba8d-94bb-4439-b991-de289b9bf66d" containerName="main"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525888    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" containerName="payload-pre-processing"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525894    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" containerName="payload-pre-processing"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525911    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2455a1c-6bdf-4458-894c-3ead12793a74" containerName="revoke-keys"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525916    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2455a1c-6bdf-4458-894c-3ead12793a74" containerName="revoke-keys"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525925    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" containerName="main"
Sep 04 23:13:03.525951 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525929    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" containerName="main"
Sep 04 23:13:03.526357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525984    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82cba8d-94bb-4439-b991-de289b9bf66d" containerName="main"
Sep 04 23:13:03.526357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525991    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="036d6b92-1f51-4bd4-86c6-8c20c9601e38" containerName="payload-pre-processing"
Sep 04 23:13:03.526357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.525998    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd38078d-f3f3-4b13-9072-a5ff0f304acf" containerName="main"
Sep 04 23:13:03.526357 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.526006    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2455a1c-6bdf-4458-894c-3ead12793a74" containerName="revoke-keys"
Sep 04 23:13:03.530305 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.530287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.532785 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.532764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 23:13:03.532926 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.532907    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-bb64b4-openshift-default-dockercfg-9sj9c\""
Sep 04 23:13:03.540551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.540525    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:13:03.596442 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqmc\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596587 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596625    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596908 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.596908 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.596770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697441 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697441 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqmc\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.697946 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.698001 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.698062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.697993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.698062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.698029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.699723 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.699702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.699883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.699866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.704934 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.704904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqmc\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.705054 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.704937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token\") pod \"e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") " pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:03.840592 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:03.840558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:04.116208 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.116128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:13:04.119718 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:13:04.119686    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6903d24_9481_44bb_9e61_5019dd39615d.slice/crio-843eb6eb9cc2600a08b56e1435fce9e6c044235690006cf330c044397f8717a9 WatchSource:0}: Error finding container 843eb6eb9cc2600a08b56e1435fce9e6c044235690006cf330c044397f8717a9: Status 404 returned error can't find the container with id 843eb6eb9cc2600a08b56e1435fce9e6c044235690006cf330c044397f8717a9
Sep 04 23:13:04.122150 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.122111    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:04.122244 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.122197    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:04.122244 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.122239    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:04.413688 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.413607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerStarted","Data":"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"}
Sep 04 23:13:04.413688 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.413657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerStarted","Data":"843eb6eb9cc2600a08b56e1435fce9e6c044235690006cf330c044397f8717a9"}
Sep 04 23:13:04.436032 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.435982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podStartSLOduration=1.435965541 podStartE2EDuration="1.435965541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:13:04.434106993 +0000 UTC m=+1668.605181607" watchObservedRunningTime="2026-09-04 23:13:04.435965541 +0000 UTC m=+1668.607040152"
Sep 04 23:13:04.841040 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:04.841008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:05.846201 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:05.846171    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:06.423721 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:06.423694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:06.424765 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:06.424726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:08.209612 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.209580    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:13:08.212627 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.212612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.214983 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.214964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-bb64b4\""
Sep 04 23:13:08.224005 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.223982    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:13:08.258759 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.258721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:13:08.261991 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.261974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.264881 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.264861    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-bb64b4-dockercfg-rt2mt\""
Sep 04 23:13:08.274795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.274772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:13:08.339256 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.339228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.339256 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.339259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrjb\" (UniqueName: \"kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.339459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.339281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7sd8\" (UniqueName: \"kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.339459 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.339351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.440528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.440496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.440707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.440577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.440707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.440597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrjb\" (UniqueName: \"kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.440707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.440618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7sd8\" (UniqueName: \"kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.441253 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.441234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.441334 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.441315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.448807 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.448774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7sd8\" (UniqueName: \"kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8\") pod \"payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.448807 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.448777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrjb\" (UniqueName: \"kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb\") pod \"payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.522940 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.522861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:08.571977 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.571946    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:08.793282 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.793254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:13:08.795657 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:13:08.795628    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe366c0_2ab2_4aea_898a_ebb400fac9ea.slice/crio-0d2d09c2da1ed15af32162c5753cd3e9d780c846de5d203fda8702da0b1dbe7f WatchSource:0}: Error finding container 0d2d09c2da1ed15af32162c5753cd3e9d780c846de5d203fda8702da0b1dbe7f: Status 404 returned error can't find the container with id 0d2d09c2da1ed15af32162c5753cd3e9d780c846de5d203fda8702da0b1dbe7f
Sep 04 23:13:08.851815 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:08.851793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:13:08.853552 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:13:08.853528    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5520e731_d57d_430a_a539_18e20fa7f420.slice/crio-a2756e020b530ea5ac7ef077d270db80038333847a2d9e9105a78dc2584e4aa6 WatchSource:0}: Error finding container a2756e020b530ea5ac7ef077d270db80038333847a2d9e9105a78dc2584e4aa6: Status 404 returned error can't find the container with id a2756e020b530ea5ac7ef077d270db80038333847a2d9e9105a78dc2584e4aa6
Sep 04 23:13:09.440277 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.440240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" event={"ID":"5520e731-d57d-430a-a539-18e20fa7f420","Type":"ContainerStarted","Data":"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"}
Sep 04 23:13:09.440803 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.440362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:09.440803 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.440388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" event={"ID":"5520e731-d57d-430a-a539-18e20fa7f420","Type":"ContainerStarted","Data":"a2756e020b530ea5ac7ef077d270db80038333847a2d9e9105a78dc2584e4aa6"}
Sep 04 23:13:09.441890 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.441866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" event={"ID":"7fe366c0-2ab2-4aea-898a-ebb400fac9ea","Type":"ContainerStarted","Data":"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"}
Sep 04 23:13:09.442017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.441912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" event={"ID":"7fe366c0-2ab2-4aea-898a-ebb400fac9ea","Type":"ContainerStarted","Data":"0d2d09c2da1ed15af32162c5753cd3e9d780c846de5d203fda8702da0b1dbe7f"}
Sep 04 23:13:09.442017 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.441946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:09.457537 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.457479    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" podStartSLOduration=1.457458278 podStartE2EDuration="1.457458278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:13:09.455346005 +0000 UTC m=+1673.626420633" watchObservedRunningTime="2026-09-04 23:13:09.457458278 +0000 UTC m=+1673.628532896"
Sep 04 23:13:09.472933 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:09.472881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" podStartSLOduration=1.47286168 podStartE2EDuration="1.47286168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:13:09.470289511 +0000 UTC m=+1673.641364123" watchObservedRunningTime="2026-09-04 23:13:09.47286168 +0000 UTC m=+1673.643936295"
Sep 04 23:13:14.590589 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:14.590553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:13:16.564632 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:16.564603    2577 scope.go:117] "RemoveContainer" containerID="c1396e6624b5401c948887baf3351e5941bda9ede5f5e399137e3f18ec39bd2f"
Sep 04 23:13:16.572895 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:16.572864    2577 scope.go:117] "RemoveContainer" containerID="9952799e931ad53ca6120d528bc8fe6ccce4010778c80806e1d7c3bdb8402dc4"
Sep 04 23:13:20.450953 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:20.450909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:13:20.452153 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:20.452131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:13:22.465927 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:22.465885    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 23:13:22.466420 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:22.465969    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 23:13:24.514799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.514663    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6903d24-9481-44bb-9e61-5019dd39615d" containerID="45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe" exitCode=0
Sep 04 23:13:24.514799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.514719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerDied","Data":"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"}
Sep 04 23:13:24.515214 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.515084    2577 scope.go:117] "RemoveContainer" containerID="45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"
Sep 04 23:13:24.516198 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.516167    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:24.516268 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.516245    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:24.516307 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:24.516280    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 04 23:13:25.521164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:25.521125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerStarted","Data":"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"}
Sep 04 23:13:25.841118 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:25.841082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:25.841478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:25.841456    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 04 23:13:25.841646 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:25.841620    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 04 23:13:26.841165 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:26.841125    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 04 23:13:26.842175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:26.841720    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 04 23:13:27.841452 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:27.841407    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 04 23:13:27.841971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:27.841479    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 04 23:13:28.841910 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:28.841868    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 04 23:13:28.842536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:28.842500    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 04 23:13:29.841872 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:29.841833    2577 patch_prober.go:28] interesting pod/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Sep 04 23:13:29.842186 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:29.842156    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Sep 04 23:13:30.861211 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:30.861181    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:30.861689 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:30.861538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:30.862712 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:30.862694    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:13:36.554562 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:36.554531    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj_5520e731-d57d-430a-a539-18e20fa7f420/payload-processing/0.log"
Sep 04 23:13:50.095593 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.095502    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:13:50.098701 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.098683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.101244 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.101219    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3543cf\"/\"openshift-service-ca.crt\""
Sep 04 23:13:50.101374 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.101336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3543cf\"/\"kube-root-ca.crt\""
Sep 04 23:13:50.102421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.102378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3543cf\"/\"default-dockercfg-vz8zl\""
Sep 04 23:13:50.102421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.102379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3543cf\"/\"ipp-route-3543cf-kserve-self-signed-certs\""
Sep 04 23:13:50.110778 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.110729    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:13:50.217996 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.217963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.218195 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.218045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.218195 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.218106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2zc\" (UniqueName: \"kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.218195 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.218174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.218346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.218208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.218346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.218245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319072 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2zc\" (UniqueName: \"kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319538 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319538 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.319687 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.319593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.321457 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.321431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.321730 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.321713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.327119 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.327099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2zc\" (UniqueName: \"kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc\") pod \"ipp-route-3543cf-kserve-787d9d67c9-lj46f\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") " pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.407925 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.407835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:13:50.678116 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:50.678020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:13:50.680835 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:13:50.680807    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc42b421_b91c_4f21_9778_431d7d2c2b01.slice/crio-042f18d09790c65f3ce7b55556e26c1e06818c9d3f75fe8cf03116f94a9f00b9 WatchSource:0}: Error finding container 042f18d09790c65f3ce7b55556e26c1e06818c9d3f75fe8cf03116f94a9f00b9: Status 404 returned error can't find the container with id 042f18d09790c65f3ce7b55556e26c1e06818c9d3f75fe8cf03116f94a9f00b9
Sep 04 23:13:51.655412 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:51.655371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" event={"ID":"fc42b421-b91c-4f21-9778-431d7d2c2b01","Type":"ContainerStarted","Data":"f8b68e80a6a82eb360ff1757607ff9a4efe5a0306747ed2b8c8025b1222af9d1"}
Sep 04 23:13:51.655412 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:51.655418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" event={"ID":"fc42b421-b91c-4f21-9778-431d7d2c2b01","Type":"ContainerStarted","Data":"042f18d09790c65f3ce7b55556e26c1e06818c9d3f75fe8cf03116f94a9f00b9"}
Sep 04 23:13:51.671816 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:51.671766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" podStartSLOduration=1.671747398 podStartE2EDuration="1.671747398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:13:51.66986324 +0000 UTC m=+1715.840937854" watchObservedRunningTime="2026-09-04 23:13:51.671747398 +0000 UTC m=+1715.842822008"
Sep 04 23:13:55.404294 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:13:55.404250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:14:00.408356 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:00.408298    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:14:00.408356 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:00.408362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:14:00.420876 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:00.420851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:14:00.706908 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:00.706820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:14:05.889438 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:05.889403    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:14:12.464475 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:12.464444    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj_5520e731-d57d-430a-a539-18e20fa7f420/payload-processing/0.log"
Sep 04 23:14:26.091851 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:26.091814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:14:51.694008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:14:51.693971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:15:00.134347 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.134312    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"]
Sep 04 23:15:00.137599 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.137576    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"]
Sep 04 23:15:00.137749 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.137706    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"
Sep 04 23:15:00.140092 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.140069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ncjwg\""
Sep 04 23:15:00.140521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.140507    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"]
Sep 04 23:15:00.140655 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.140637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"
Sep 04 23:15:00.144419 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.144399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"]
Sep 04 23:15:00.144528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.144493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"
Sep 04 23:15:00.146207 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.146186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"]
Sep 04 23:15:00.151158 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.151140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"
Sep 04 23:15:00.160895 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.160869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"
Sep 04 23:15:00.161164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.160900    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"]
Sep 04 23:15:00.190428 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.190401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"
Sep 04 23:15:00.435768 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.435683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"]
Sep 04 23:15:00.439075 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:15:00.439051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab49d14_682d_4065_b394_7856153dad4b.slice/crio-dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648 WatchSource:0}: Error finding container dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648: Status 404 returned error can't find the container with id dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648
Sep 04 23:15:00.449224 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.449200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"]
Sep 04 23:15:00.452679 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:15:00.452657    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08be0236_7591_4985_8087_ac9939daac1d.slice/crio-a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f WatchSource:0}: Error finding container a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f: Status 404 returned error can't find the container with id a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f
Sep 04 23:15:00.474886 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.474864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"]
Sep 04 23:15:00.477134 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:15:00.477110    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedcc63f_f958_4da6_babe_19bc0aa4b5ee.slice/crio-a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1 WatchSource:0}: Error finding container a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1: Status 404 returned error can't find the container with id a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1
Sep 04 23:15:00.964524 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.964483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z" event={"ID":"08be0236-7591-4985-8087-ac9939daac1d","Type":"ContainerStarted","Data":"d76b89624f782b679eb1c7cce95d94f1203d2aee5465dbe7709d8e4912db233b"}
Sep 04 23:15:00.964715 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.964586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z" event={"ID":"08be0236-7591-4985-8087-ac9939daac1d","Type":"ContainerStarted","Data":"a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f"}
Sep 04 23:15:00.966054 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.966023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds" event={"ID":"dab49d14-682d-4065-b394-7856153dad4b","Type":"ContainerStarted","Data":"ab0d9b962c7fc4e930beb3b94c3fbaf49cd5c8e7337d89555ef1407400587f66"}
Sep 04 23:15:00.966175 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.966075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds" event={"ID":"dab49d14-682d-4065-b394-7856153dad4b","Type":"ContainerStarted","Data":"dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648"}
Sep 04 23:15:00.967442 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.967420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv" event={"ID":"dedcc63f-f958-4da6-babe-19bc0aa4b5ee","Type":"ContainerStarted","Data":"e3911f70ef868716cda4b4b57d2f6eb3b63778398f7270518f6b1af8294ef896"}
Sep 04 23:15:00.967536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.967458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv" event={"ID":"dedcc63f-f958-4da6-babe-19bc0aa4b5ee","Type":"ContainerStarted","Data":"a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1"}
Sep 04 23:15:00.979208 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.979169    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z" podStartSLOduration=0.979156405 podStartE2EDuration="979.156405ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:15:00.978370469 +0000 UTC m=+1785.149445081" watchObservedRunningTime="2026-09-04 23:15:00.979156405 +0000 UTC m=+1785.150231017"
Sep 04 23:15:00.991698 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:00.991655    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds" podStartSLOduration=0.991643826 podStartE2EDuration="991.643826ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:15:00.991421842 +0000 UTC m=+1785.162496456" watchObservedRunningTime="2026-09-04 23:15:00.991643826 +0000 UTC m=+1785.162718429"
Sep 04 23:15:01.007381 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.007300    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv" podStartSLOduration=1.007287256 podStartE2EDuration="1.007287256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:15:01.005514875 +0000 UTC m=+1785.176589499" watchObservedRunningTime="2026-09-04 23:15:01.007287256 +0000 UTC m=+1785.178361892"
Sep 04 23:15:01.973580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.973485    2577 generic.go:356] "Generic (PLEG): container finished" podID="08be0236-7591-4985-8087-ac9939daac1d" containerID="d76b89624f782b679eb1c7cce95d94f1203d2aee5465dbe7709d8e4912db233b" exitCode=0
Sep 04 23:15:01.973580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.973546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z" event={"ID":"08be0236-7591-4985-8087-ac9939daac1d","Type":"ContainerDied","Data":"d76b89624f782b679eb1c7cce95d94f1203d2aee5465dbe7709d8e4912db233b"}
Sep 04 23:15:01.975114 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.975089    2577 generic.go:356] "Generic (PLEG): container finished" podID="dab49d14-682d-4065-b394-7856153dad4b" containerID="ab0d9b962c7fc4e930beb3b94c3fbaf49cd5c8e7337d89555ef1407400587f66" exitCode=0
Sep 04 23:15:01.975210 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.975158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds" event={"ID":"dab49d14-682d-4065-b394-7856153dad4b","Type":"ContainerDied","Data":"ab0d9b962c7fc4e930beb3b94c3fbaf49cd5c8e7337d89555ef1407400587f66"}
Sep 04 23:15:01.976766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.976725    2577 generic.go:356] "Generic (PLEG): container finished" podID="dedcc63f-f958-4da6-babe-19bc0aa4b5ee" containerID="e3911f70ef868716cda4b4b57d2f6eb3b63778398f7270518f6b1af8294ef896" exitCode=0
Sep 04 23:15:01.976766 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:01.976757    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv" event={"ID":"dedcc63f-f958-4da6-babe-19bc0aa4b5ee","Type":"ContainerDied","Data":"e3911f70ef868716cda4b4b57d2f6eb3b63778398f7270518f6b1af8294ef896"}
Sep 04 23:15:03.133336 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.133306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"
Sep 04 23:15:03.171062 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.171026    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"
Sep 04 23:15:03.173987 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.173970    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"
Sep 04 23:15:03.986423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.986392    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"
Sep 04 23:15:03.986628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.986432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds" event={"ID":"dab49d14-682d-4065-b394-7856153dad4b","Type":"ContainerDied","Data":"dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648"}
Sep 04 23:15:03.986628 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.986479    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd68995019e7bfbbb52a4a67a430e20bf532c9305b0c9ad8eda0eccb3147f648"
Sep 04 23:15:03.987790 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.987765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv" event={"ID":"dedcc63f-f958-4da6-babe-19bc0aa4b5ee","Type":"ContainerDied","Data":"a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1"}
Sep 04 23:15:03.987923 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.987807    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34a4cf157b7d6a3c38408cc2c203b439c5d2c38b133e4e15a4b7b332693afe1"
Sep 04 23:15:03.987923 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.987774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"
Sep 04 23:15:03.989020 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.989002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z" event={"ID":"08be0236-7591-4985-8087-ac9939daac1d","Type":"ContainerDied","Data":"a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f"}
Sep 04 23:15:03.989106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.989035    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19307ac0704277d37b1e655856150d8742d8d5ea4789c0dadd2fe0f8d9cef3f"
Sep 04 23:15:03.989106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:03.989034    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"
Sep 04 23:15:14.302708 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.302624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"]
Sep 04 23:15:14.304272 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.304250    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3543cf-29809395-jg8ds"]
Sep 04 23:15:14.326624 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.326591    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab49d14-682d-4065-b394-7856153dad4b" path="/var/lib/kubelet/pods/dab49d14-682d-4065-b394-7856153dad4b/volumes"
Sep 04 23:15:14.339180 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.339124    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:15:14.339449 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.339402    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" podUID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" containerName="payload-pre-processing" containerID="cri-o://17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6" gracePeriod=30
Sep 04 23:15:14.587296 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.587269    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:15:14.671332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.671247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nntm4\" (UniqueName: \"kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4\") pod \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") "
Sep 04 23:15:14.671539 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.671412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume\") pod \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\" (UID: \"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5\") "
Sep 04 23:15:14.671876 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.671843    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" (UID: "ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:15:14.673568 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.673541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4" (OuterVolumeSpecName: "kube-api-access-nntm4") pod "ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" (UID: "ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5"). InnerVolumeSpecName "kube-api-access-nntm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:15:14.772663 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.772633    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntm4\" (UniqueName: \"kubernetes.io/projected/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-kube-api-access-nntm4\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:14.772663 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:14.772658    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:15.041168 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.041132    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" containerID="17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6" exitCode=0
Sep 04 23:15:15.041323 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.041176    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"
Sep 04 23:15:15.041323 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.041219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" event={"ID":"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5","Type":"ContainerDied","Data":"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"}
Sep 04 23:15:15.041323 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.041267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth" event={"ID":"ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5","Type":"ContainerDied","Data":"1168a05f2829fd27ba720ff8c2158ba746666213107234588350c3cb1991059e"}
Sep 04 23:15:15.041323 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.041291    2577 scope.go:117] "RemoveContainer" containerID="17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"
Sep 04 23:15:15.050354 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.050340    2577 scope.go:117] "RemoveContainer" containerID="17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"
Sep 04 23:15:15.050613 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:15:15.050596    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6\": container with ID starting with 17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6 not found: ID does not exist" containerID="17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"
Sep 04 23:15:15.050658 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.050620    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6"} err="failed to get container status \"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6\": rpc error: code = NotFound desc = could not find container \"17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6\": container with ID starting with 17e039101fc54515d0ccdbc4498324f24707076d2c093c811bb3d0f95cc909f6 not found: ID does not exist"
Sep 04 23:15:15.061654 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.061634    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:15:15.064277 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.064253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3543cf-bd665b946-grfth"]
Sep 04 23:15:15.097754 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:15.097703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:15:16.328593 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:16.328559    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" path="/var/lib/kubelet/pods/ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5/volumes"
Sep 04 23:15:16.618753 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:16.618708    2577 scope.go:117] "RemoveContainer" containerID="1e8b02e73e9e6b005039f4ace24f61066b5b6751714c1f6a5f016854d9909464"
Sep 04 23:15:19.892756 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:19.892707    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:15:24.193845 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:24.193810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:15:34.912713 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:34.912673    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" podUID="fc42b421-b91c-4f21-9778-431d7d2c2b01" containerName="main" containerID="cri-o://f8b68e80a6a82eb360ff1757607ff9a4efe5a0306747ed2b8c8025b1222af9d1" gracePeriod=45
Sep 04 23:15:35.136429 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.136397    2577 generic.go:356] "Generic (PLEG): container finished" podID="fc42b421-b91c-4f21-9778-431d7d2c2b01" containerID="f8b68e80a6a82eb360ff1757607ff9a4efe5a0306747ed2b8c8025b1222af9d1" exitCode=0
Sep 04 23:15:35.136583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.136466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" event={"ID":"fc42b421-b91c-4f21-9778-431d7d2c2b01","Type":"ContainerDied","Data":"f8b68e80a6a82eb360ff1757607ff9a4efe5a0306747ed2b8c8025b1222af9d1"}
Sep 04 23:15:35.161855 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.161831    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:15:35.261561 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261561 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261521    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp2zc\" (UniqueName: \"kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261561 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261566    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261631    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs\") pod \"fc42b421-b91c-4f21-9778-431d7d2c2b01\" (UID: \"fc42b421-b91c-4f21-9778-431d7d2c2b01\") "
Sep 04 23:15:35.261880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261805    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home" (OuterVolumeSpecName: "home") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:15:35.262074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261906    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:15:35.262074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:15:35.262074 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.261969    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:35.263799 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.263778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm" (OuterVolumeSpecName: "dshm") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:15:35.263883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.263836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc" (OuterVolumeSpecName: "kube-api-access-gp2zc") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "kube-api-access-gp2zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:15:35.263883 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.263841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc42b421-b91c-4f21-9778-431d7d2c2b01" (UID: "fc42b421-b91c-4f21-9778-431d7d2c2b01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:15:35.363423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.363384    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:35.363423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.363414    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc42b421-b91c-4f21-9778-431d7d2c2b01-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:35.363423 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.363424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp2zc\" (UniqueName: \"kubernetes.io/projected/fc42b421-b91c-4f21-9778-431d7d2c2b01-kube-api-access-gp2zc\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:35.363663 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.363433    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:35.363663 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:35.363443    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc42b421-b91c-4f21-9778-431d7d2c2b01-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:15:36.142065 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.142024    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"
Sep 04 23:15:36.142533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.142037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f" event={"ID":"fc42b421-b91c-4f21-9778-431d7d2c2b01","Type":"ContainerDied","Data":"042f18d09790c65f3ce7b55556e26c1e06818c9d3f75fe8cf03116f94a9f00b9"}
Sep 04 23:15:36.142533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.142160    2577 scope.go:117] "RemoveContainer" containerID="f8b68e80a6a82eb360ff1757607ff9a4efe5a0306747ed2b8c8025b1222af9d1"
Sep 04 23:15:36.162392 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.162368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:15:36.166199 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.166172    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3543cf/ipp-route-3543cf-kserve-787d9d67c9-lj46f"]
Sep 04 23:15:36.329950 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:15:36.327087    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc42b421-b91c-4f21-9778-431d7d2c2b01" path="/var/lib/kubelet/pods/fc42b421-b91c-4f21-9778-431d7d2c2b01/volumes"
Sep 04 23:16:13.401802 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.401768    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"]
Sep 04 23:16:13.411121 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.411092    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-bb64b4-29809395-pc29z"]
Sep 04 23:16:13.464240 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.464203    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:16:13.464510 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.464466    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" podUID="5520e731-d57d-430a-a539-18e20fa7f420" containerName="payload-processing" containerID="cri-o://1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363" gracePeriod=30
Sep 04 23:16:13.472188 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.472163    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:16:13.472404 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.472380    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" podUID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" containerName="payload-pre-processing" containerID="cri-o://bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657" gracePeriod=30
Sep 04 23:16:13.745097 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.745074    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:16:13.748242 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.748222    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:16:13.807311 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7sd8\" (UniqueName: \"kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8\") pod \"5520e731-d57d-430a-a539-18e20fa7f420\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") "
Sep 04 23:16:13.807488 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume\") pod \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") "
Sep 04 23:16:13.807488 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume\") pod \"5520e731-d57d-430a-a539-18e20fa7f420\" (UID: \"5520e731-d57d-430a-a539-18e20fa7f420\") "
Sep 04 23:16:13.807488 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqrjb\" (UniqueName: \"kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb\") pod \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\" (UID: \"7fe366c0-2ab2-4aea-898a-ebb400fac9ea\") "
Sep 04 23:16:13.807812 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5520e731-d57d-430a-a539-18e20fa7f420" (UID: "5520e731-d57d-430a-a539-18e20fa7f420"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:13.807812 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.807798    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fe366c0-2ab2-4aea-898a-ebb400fac9ea" (UID: "7fe366c0-2ab2-4aea-898a-ebb400fac9ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:13.809370 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.809348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8" (OuterVolumeSpecName: "kube-api-access-r7sd8") pod "5520e731-d57d-430a-a539-18e20fa7f420" (UID: "5520e731-d57d-430a-a539-18e20fa7f420"). InnerVolumeSpecName "kube-api-access-r7sd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:13.809874 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.809847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb" (OuterVolumeSpecName: "kube-api-access-tqrjb") pod "7fe366c0-2ab2-4aea-898a-ebb400fac9ea" (UID: "7fe366c0-2ab2-4aea-898a-ebb400fac9ea"). InnerVolumeSpecName "kube-api-access-tqrjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:13.908333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.908296    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7sd8\" (UniqueName: \"kubernetes.io/projected/5520e731-d57d-430a-a539-18e20fa7f420-kube-api-access-r7sd8\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:13.908333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.908327    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:13.908333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.908337    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5520e731-d57d-430a-a539-18e20fa7f420-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:13.908552 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:13.908345    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrjb\" (UniqueName: \"kubernetes.io/projected/7fe366c0-2ab2-4aea-898a-ebb400fac9ea-kube-api-access-tqrjb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:14.320094 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.320062    2577 generic.go:356] "Generic (PLEG): container finished" podID="5520e731-d57d-430a-a539-18e20fa7f420" containerID="1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363" exitCode=0
Sep 04 23:16:14.320285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.320172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"
Sep 04 23:16:14.321548 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.321522    2577 generic.go:356] "Generic (PLEG): container finished" podID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" containerID="bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657" exitCode=0
Sep 04 23:16:14.321670 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.321631    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"
Sep 04 23:16:14.324685 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.324663    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08be0236-7591-4985-8087-ac9939daac1d" path="/var/lib/kubelet/pods/08be0236-7591-4985-8087-ac9939daac1d/volumes"
Sep 04 23:16:14.325093 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.325072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" event={"ID":"5520e731-d57d-430a-a539-18e20fa7f420","Type":"ContainerDied","Data":"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"}
Sep 04 23:16:14.325194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.325109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj" event={"ID":"5520e731-d57d-430a-a539-18e20fa7f420","Type":"ContainerDied","Data":"a2756e020b530ea5ac7ef077d270db80038333847a2d9e9105a78dc2584e4aa6"}
Sep 04 23:16:14.325194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.325134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" event={"ID":"7fe366c0-2ab2-4aea-898a-ebb400fac9ea","Type":"ContainerDied","Data":"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"}
Sep 04 23:16:14.325194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.325158    2577 scope.go:117] "RemoveContainer" containerID="1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"
Sep 04 23:16:14.325336 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.325159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb" event={"ID":"7fe366c0-2ab2-4aea-898a-ebb400fac9ea","Type":"ContainerDied","Data":"0d2d09c2da1ed15af32162c5753cd3e9d780c846de5d203fda8702da0b1dbe7f"}
Sep 04 23:16:14.334929 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.334881    2577 scope.go:117] "RemoveContainer" containerID="1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"
Sep 04 23:16:14.335240 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:14.335221    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363\": container with ID starting with 1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363 not found: ID does not exist" containerID="1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"
Sep 04 23:16:14.335329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.335246    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363"} err="failed to get container status \"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363\": rpc error: code = NotFound desc = could not find container \"1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363\": container with ID starting with 1160da0e73baf26750825bcfbf0181d4a4bfe6db6ca74c815b63757a53812363 not found: ID does not exist"
Sep 04 23:16:14.335329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.335262    2577 scope.go:117] "RemoveContainer" containerID="bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"
Sep 04 23:16:14.343283 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.343266    2577 scope.go:117] "RemoveContainer" containerID="bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"
Sep 04 23:16:14.343549 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:14.343530    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657\": container with ID starting with bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657 not found: ID does not exist" containerID="bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"
Sep 04 23:16:14.343593 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.343558    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657"} err="failed to get container status \"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657\": rpc error: code = NotFound desc = could not find container \"bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657\": container with ID starting with bdc113acb8a7c94df0ba610a1d3a16b35bb9842f13f71a79e78966d983a21657 not found: ID does not exist"
Sep 04 23:16:14.347699 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.347669    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:16:14.350001 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.349982    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-bb64b4-6b45ff4d4c-9bxsj"]
Sep 04 23:16:14.359779 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.359716    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:16:14.363536 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:14.363517    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-bb64b4-9b8875c46-kvtcb"]
Sep 04 23:16:16.325341 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:16.325301    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5520e731-d57d-430a-a539-18e20fa7f420" path="/var/lib/kubelet/pods/5520e731-d57d-430a-a539-18e20fa7f420/volumes"
Sep 04 23:16:16.325859 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:16.325837    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" path="/var/lib/kubelet/pods/7fe366c0-2ab2-4aea-898a-ebb400fac9ea/volumes"
Sep 04 23:16:18.094555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:18.094522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:16:19.957136 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:19.957106    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:16:19.957538 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:19.957308    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy" containerID="cri-o://befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a" gracePeriod=30
Sep 04 23:16:25.198008 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.197980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:16:25.310771 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310647    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.310771 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310696    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.310771 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310754    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkqmc\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.310771 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.310986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:16:25.311115 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy\") pod \"d6903d24-9481-44bb-9e61-5019dd39615d\" (UID: \"d6903d24-9481-44bb-9e61-5019dd39615d\") "
Sep 04 23:16:25.311421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311185    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:25.311421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311274    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:16:25.311421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311330    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-credential-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.311575 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311413    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:16:25.311575 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.311476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:16:25.313086 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.313056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 23:16:25.313194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.313116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:16:25.313194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.313146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc" (OuterVolumeSpecName: "kube-api-access-pkqmc") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "kube-api-access-pkqmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:25.313194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.313177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6903d24-9481-44bb-9e61-5019dd39615d" (UID: "d6903d24-9481-44bb-9e61-5019dd39615d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:25.373987 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.373949    2577 generic.go:356] "Generic (PLEG): container finished" podID="d6903d24-9481-44bb-9e61-5019dd39615d" containerID="befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a" exitCode=0
Sep 04 23:16:25.373987 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.373992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"
Sep 04 23:16:25.374201 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.374027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerDied","Data":"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"}
Sep 04 23:16:25.374201 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.374098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk" event={"ID":"d6903d24-9481-44bb-9e61-5019dd39615d","Type":"ContainerDied","Data":"843eb6eb9cc2600a08b56e1435fce9e6c044235690006cf330c044397f8717a9"}
Sep 04 23:16:25.374201 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.374122    2577 scope.go:117] "RemoveContainer" containerID="befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"
Sep 04 23:16:25.383780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.383760    2577 scope.go:117] "RemoveContainer" containerID="45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"
Sep 04 23:16:25.391698 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.391682    2577 scope.go:117] "RemoveContainer" containerID="befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"
Sep 04 23:16:25.391999 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:25.391981    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a\": container with ID starting with befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a not found: ID does not exist" containerID="befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"
Sep 04 23:16:25.392060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.392010    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a"} err="failed to get container status \"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a\": rpc error: code = NotFound desc = could not find container \"befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a\": container with ID starting with befcd50e4e52ef626b381478e5a79c5c5ed5e1e1bd987e9b0d0f46f449045c0a not found: ID does not exist"
Sep 04 23:16:25.392060 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.392027    2577 scope.go:117] "RemoveContainer" containerID="45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"
Sep 04 23:16:25.392275 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:25.392254    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe\": container with ID starting with 45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe not found: ID does not exist" containerID="45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"
Sep 04 23:16:25.392331 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.392285    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe"} err="failed to get container status \"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe\": rpc error: code = NotFound desc = could not find container \"45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe\": container with ID starting with 45ad807124a6293a1760de614f7401e5d29f8ce3e4a37317549066c997617dbe not found: ID does not exist"
Sep 04 23:16:25.395984 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.395963    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:16:25.400806 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.400786    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-bb64b4-openshift-default-6b8bc599fb-7xrkk"]
Sep 04 23:16:25.412533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412509    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-data\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412533 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412529    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-istio-token\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412542    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-istio-envoy\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412555    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6903d24-9481-44bb-9e61-5019dd39615d-istiod-ca-cert\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412567    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412581    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqmc\" (UniqueName: \"kubernetes.io/projected/d6903d24-9481-44bb-9e61-5019dd39615d-kube-api-access-pkqmc\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412595    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6903d24-9481-44bb-9e61-5019dd39615d-workload-socket\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:25.412667 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:25.412609    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6903d24-9481-44bb-9e61-5019dd39615d-istio-podinfo\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:26.324880 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:26.324848    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" path="/var/lib/kubelet/pods/d6903d24-9481-44bb-9e61-5019dd39615d/volumes"
Sep 04 23:16:30.772691 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.772661    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"]
Sep 04 23:16:30.778003 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.777976    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9ec1e5-29809395-2hwkv"]
Sep 04 23:16:30.802319 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.802282    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:16:30.802541 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.802507    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" podUID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" containerName="payload-processing" containerID="cri-o://2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18" gracePeriod=30
Sep 04 23:16:30.809299 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.809267    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:16:30.809704 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:30.809670    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" podUID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" containerName="payload-pre-processing" containerID="cri-o://4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b" gracePeriod=30
Sep 04 23:16:31.061813 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.061784    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:16:31.085238 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.085216    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:16:31.159577 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.159542    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9blsb\" (UniqueName: \"kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb\") pod \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") "
Sep 04 23:16:31.159776 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.159616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume\") pod \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\" (UID: \"c6acfd2e-b4d5-4300-8ea1-f0c21635d625\") "
Sep 04 23:16:31.160010 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.159978    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6acfd2e-b4d5-4300-8ea1-f0c21635d625" (UID: "c6acfd2e-b4d5-4300-8ea1-f0c21635d625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:31.161682 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.161660    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb" (OuterVolumeSpecName: "kube-api-access-9blsb") pod "c6acfd2e-b4d5-4300-8ea1-f0c21635d625" (UID: "c6acfd2e-b4d5-4300-8ea1-f0c21635d625"). InnerVolumeSpecName "kube-api-access-9blsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:31.260346 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.260311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume\") pod \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") "
Sep 04 23:16:31.260523 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.260364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22vmq\" (UniqueName: \"kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq\") pod \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\" (UID: \"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb\") "
Sep 04 23:16:31.260580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.260532    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blsb\" (UniqueName: \"kubernetes.io/projected/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-kube-api-access-9blsb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:31.260580 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.260543    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acfd2e-b4d5-4300-8ea1-f0c21635d625-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:31.260681 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.260657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" (UID: "e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:31.262386 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.262366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq" (OuterVolumeSpecName: "kube-api-access-22vmq") pod "e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" (UID: "e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb"). InnerVolumeSpecName "kube-api-access-22vmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:31.361243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.361205    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:31.361243 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.361236    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22vmq\" (UniqueName: \"kubernetes.io/projected/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb-kube-api-access-22vmq\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:31.403058 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.403025    2577 generic.go:356] "Generic (PLEG): container finished" podID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" containerID="2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18" exitCode=0
Sep 04 23:16:31.403185 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.403070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"
Sep 04 23:16:31.403185 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.403108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" event={"ID":"c6acfd2e-b4d5-4300-8ea1-f0c21635d625","Type":"ContainerDied","Data":"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"}
Sep 04 23:16:31.403185 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.403164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk" event={"ID":"c6acfd2e-b4d5-4300-8ea1-f0c21635d625","Type":"ContainerDied","Data":"05f989852c8a4ee3da34380e3e81807b1b484eba46903bb01e6670b0429f8585"}
Sep 04 23:16:31.403332 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.403192    2577 scope.go:117] "RemoveContainer" containerID="2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"
Sep 04 23:16:31.404456 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.404433    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" containerID="4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b" exitCode=0
Sep 04 23:16:31.404553 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.404463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" event={"ID":"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb","Type":"ContainerDied","Data":"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"}
Sep 04 23:16:31.404553 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.404481    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"
Sep 04 23:16:31.404553 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.404495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7" event={"ID":"e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb","Type":"ContainerDied","Data":"4518e7fecf88e149665cd59fcf2de18af94e7817a910c6344666ff7a563ded15"}
Sep 04 23:16:31.412188 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.412166    2577 scope.go:117] "RemoveContainer" containerID="2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"
Sep 04 23:16:31.412488 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:31.412463    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18\": container with ID starting with 2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18 not found: ID does not exist" containerID="2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"
Sep 04 23:16:31.412548 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.412500    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18"} err="failed to get container status \"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18\": rpc error: code = NotFound desc = could not find container \"2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18\": container with ID starting with 2c611446128a0635f1a8893e2b278b23041773dda28c7e1cb61fe1aa9e4afc18 not found: ID does not exist"
Sep 04 23:16:31.412548 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.412525    2577 scope.go:117] "RemoveContainer" containerID="4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"
Sep 04 23:16:31.420059 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.420041    2577 scope.go:117] "RemoveContainer" containerID="4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"
Sep 04 23:16:31.420344 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:16:31.420323    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b\": container with ID starting with 4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b not found: ID does not exist" containerID="4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"
Sep 04 23:16:31.420424 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.420350    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b"} err="failed to get container status \"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b\": rpc error: code = NotFound desc = could not find container \"4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b\": container with ID starting with 4f0d65c23c7d6bb24bb3d0ac7fe4e979d82a26abbf8382bfbde815cf536deb2b not found: ID does not exist"
Sep 04 23:16:31.429356 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.429328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:16:31.434949 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.434927    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-9ec1e5-7ff5d6c455-ghhhk"]
Sep 04 23:16:31.444659 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.444637    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:16:31.450197 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:31.450179    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-9ec1e5-8696dbbd4b-ggbr7"]
Sep 04 23:16:32.324794 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:32.324760    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" path="/var/lib/kubelet/pods/c6acfd2e-b4d5-4300-8ea1-f0c21635d625/volumes"
Sep 04 23:16:32.325173 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:32.325042    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dedcc63f-f958-4da6-babe-19bc0aa4b5ee" path="/var/lib/kubelet/pods/dedcc63f-f958-4da6-babe-19bc0aa4b5ee/volumes"
Sep 04 23:16:32.325315 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:32.325302    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" path="/var/lib/kubelet/pods/e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb/volumes"
Sep 04 23:16:34.187433 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:34.187392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:16:48.263098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.263019    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:16:48.263634 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.263282    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" podUID="951b4db4-b28e-436f-8397-b97404d6477c" containerName="payload-processing" containerID="cri-o://c833cec0a22d9b5ed78a4798a1998f441c91a2cb18bb4e5cbbf92ac2a832a901" gracePeriod=30
Sep 04 23:16:48.270042 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.270007    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:16:48.270466 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.270202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" podUID="459f95d2-432b-4601-b4d8-1630e0431164" containerName="payload-pre-processing" containerID="cri-o://eec905c0d7e254ee8d59644397adcfd524f4a6f366ff8294b83b76b61eca95c4" gracePeriod=30
Sep 04 23:16:48.483306 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.483271    2577 generic.go:356] "Generic (PLEG): container finished" podID="951b4db4-b28e-436f-8397-b97404d6477c" containerID="c833cec0a22d9b5ed78a4798a1998f441c91a2cb18bb4e5cbbf92ac2a832a901" exitCode=0
Sep 04 23:16:48.483478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.483345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" event={"ID":"951b4db4-b28e-436f-8397-b97404d6477c","Type":"ContainerDied","Data":"c833cec0a22d9b5ed78a4798a1998f441c91a2cb18bb4e5cbbf92ac2a832a901"}
Sep 04 23:16:48.485235 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.485213    2577 generic.go:356] "Generic (PLEG): container finished" podID="459f95d2-432b-4601-b4d8-1630e0431164" containerID="eec905c0d7e254ee8d59644397adcfd524f4a6f366ff8294b83b76b61eca95c4" exitCode=0
Sep 04 23:16:48.485352 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.485283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" event={"ID":"459f95d2-432b-4601-b4d8-1630e0431164","Type":"ContainerDied","Data":"eec905c0d7e254ee8d59644397adcfd524f4a6f366ff8294b83b76b61eca95c4"}
Sep 04 23:16:48.533792 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.533724    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:16:48.536641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.536621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:16:48.597971 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.597935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz2gb\" (UniqueName: \"kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb\") pod \"459f95d2-432b-4601-b4d8-1630e0431164\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") "
Sep 04 23:16:48.598163 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.597983    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n78sh\" (UniqueName: \"kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh\") pod \"951b4db4-b28e-436f-8397-b97404d6477c\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") "
Sep 04 23:16:48.598163 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.598003    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume\") pod \"951b4db4-b28e-436f-8397-b97404d6477c\" (UID: \"951b4db4-b28e-436f-8397-b97404d6477c\") "
Sep 04 23:16:48.598163 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.598043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume\") pod \"459f95d2-432b-4601-b4d8-1630e0431164\" (UID: \"459f95d2-432b-4601-b4d8-1630e0431164\") "
Sep 04 23:16:48.598372 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.598351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "951b4db4-b28e-436f-8397-b97404d6477c" (UID: "951b4db4-b28e-436f-8397-b97404d6477c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:48.598479 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.598456    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "459f95d2-432b-4601-b4d8-1630e0431164" (UID: "459f95d2-432b-4601-b4d8-1630e0431164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 23:16:48.600027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.600003    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb" (OuterVolumeSpecName: "kube-api-access-pz2gb") pod "459f95d2-432b-4601-b4d8-1630e0431164" (UID: "459f95d2-432b-4601-b4d8-1630e0431164"). InnerVolumeSpecName "kube-api-access-pz2gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:48.600027 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.600011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh" (OuterVolumeSpecName: "kube-api-access-n78sh") pod "951b4db4-b28e-436f-8397-b97404d6477c" (UID: "951b4db4-b28e-436f-8397-b97404d6477c"). InnerVolumeSpecName "kube-api-access-n78sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:16:48.699324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.699290    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459f95d2-432b-4601-b4d8-1630e0431164-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:48.699324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.699316    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2gb\" (UniqueName: \"kubernetes.io/projected/459f95d2-432b-4601-b4d8-1630e0431164-kube-api-access-pz2gb\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:48.699324 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.699326    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78sh\" (UniqueName: \"kubernetes.io/projected/951b4db4-b28e-436f-8397-b97404d6477c-kube-api-access-n78sh\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:48.699540 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:48.699336    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951b4db4-b28e-436f-8397-b97404d6477c-plugins-config-volume\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:16:49.490838 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.490807    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"
Sep 04 23:16:49.491284 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.490804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls" event={"ID":"951b4db4-b28e-436f-8397-b97404d6477c","Type":"ContainerDied","Data":"df2e6d609b44aad2d8dac7b224a84e717254da2580b633a5ca6e6850538c1692"}
Sep 04 23:16:49.491284 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.490961    2577 scope.go:117] "RemoveContainer" containerID="c833cec0a22d9b5ed78a4798a1998f441c91a2cb18bb4e5cbbf92ac2a832a901"
Sep 04 23:16:49.492333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.492181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9" event={"ID":"459f95d2-432b-4601-b4d8-1630e0431164","Type":"ContainerDied","Data":"2ed96abb65f298b1ba9d4be6985e7884f6c3d3f3df482269016e9d87866c5681"}
Sep 04 23:16:49.492333 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.492252    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"
Sep 04 23:16:49.501810 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.501781    2577 scope.go:117] "RemoveContainer" containerID="eec905c0d7e254ee8d59644397adcfd524f4a6f366ff8294b83b76b61eca95c4"
Sep 04 23:16:49.537652 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.537597    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:16:49.551712 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.551687    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-146c0e-565d679794-5vkp9"]
Sep 04 23:16:49.565633 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.565612    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:16:49.571047 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:49.571028    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-146c0e-d754d8999-8dxls"]
Sep 04 23:16:50.325421 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:50.325386    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459f95d2-432b-4601-b4d8-1630e0431164" path="/var/lib/kubelet/pods/459f95d2-432b-4601-b4d8-1630e0431164/volumes"
Sep 04 23:16:50.325749 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:50.325726    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951b4db4-b28e-436f-8397-b97404d6477c" path="/var/lib/kubelet/pods/951b4db4-b28e-436f-8397-b97404d6477c/volumes"
Sep 04 23:16:51.901170 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:51.901135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:16:59.492932 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:16:59.492887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:18:16.680385 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:18:16.680310    2577 scope.go:117] "RemoveContainer" containerID="53863b4fa0e77f46011f59c7235dca9675ac25dc28391f8941a0d93b9e59db04"
Sep 04 23:18:20.293260 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:18:20.293225    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:18:48.697521 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:18:48.697485    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:18:55.491329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:18:55.491295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:19:01.584612 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:19:01.584574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:19:08.590164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:19:08.590126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:19:16.697555 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:19:16.697521    2577 scope.go:117] "RemoveContainer" containerID="857ce4c06c5b769097df85bbd4514c06cac82af725bcf1f706558a7786d06b6f"
Sep 04 23:19:26.388465 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:19:26.388431    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:19:33.519716 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:19:33.519678    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:01.387642 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:01.387588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:08.784978 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:08.784891    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:11.079777 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.079722    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080032    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951b4db4-b28e-436f-8397-b97404d6477c" containerName="payload-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080044    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="951b4db4-b28e-436f-8397-b97404d6477c" containerName="payload-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080054    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dedcc63f-f958-4da6-babe-19bc0aa4b5ee" containerName="cleanup"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080060    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dedcc63f-f958-4da6-babe-19bc0aa4b5ee" containerName="cleanup"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080067    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080074    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" containerName="payload-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" containerName="payload-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080101    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc42b421-b91c-4f21-9778-431d7d2c2b01" containerName="main"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080107    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc42b421-b91c-4f21-9778-431d7d2c2b01" containerName="main"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080116    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080121    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080128    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080134    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080140    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080145    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" containerName="payload-pre-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080156    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5520e731-d57d-430a-a539-18e20fa7f420" containerName="payload-processing"
Sep 04 23:21:11.080164 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5520e731-d57d-430a-a539-18e20fa7f420" containerName="payload-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080177    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08be0236-7591-4985-8087-ac9939daac1d" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080183    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08be0236-7591-4985-8087-ac9939daac1d" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080193    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459f95d2-432b-4601-b4d8-1630e0431164" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080198    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="459f95d2-432b-4601-b4d8-1630e0431164" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080204    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab49d14-682d-4065-b394-7856153dad4b" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab49d14-682d-4065-b394-7856153dad4b" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080258    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd3cd9a-c9d0-42aa-b3ce-8562752f9cc5" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080265    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="951b4db4-b28e-436f-8397-b97404d6477c" containerName="payload-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080270    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5520e731-d57d-430a-a539-18e20fa7f420" containerName="payload-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080275    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dedcc63f-f958-4da6-babe-19bc0aa4b5ee" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080280    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6acfd2e-b4d5-4300-8ea1-f0c21635d625" containerName="payload-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080285    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08be0236-7591-4985-8087-ac9939daac1d" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080291    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ded5a1-c745-4b9a-92a6-7057d8c0f3bb" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080297    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc42b421-b91c-4f21-9778-431d7d2c2b01" containerName="main"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080301    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab49d14-682d-4065-b394-7856153dad4b" containerName="cleanup"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080305    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe366c0-2ab2-4aea-898a-ebb400fac9ea" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080311    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="459f95d2-432b-4601-b4d8-1630e0431164" containerName="payload-pre-processing"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080317    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:11.080707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.080324    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:11.083183 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.083162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.085531 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.085505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-24815903-kserve-self-signed-certs\""
Sep 04 23:21:11.091215 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.091189    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:11.170781 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170747    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.170968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.170968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.170968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.170968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.171126 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.170969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwhm\" (UniqueName: \"kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272395 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272395 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272641 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwhm\" (UniqueName: \"kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.272934 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.273051 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.273051 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.272982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.274725 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.274704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.275018 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.275001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.283652 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.283625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwhm\" (UniqueName: \"kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm\") pod \"e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") " pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.392019 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.391975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:11.660528 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:11.660435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:11.664660 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:21:11.664637    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d0aaa25_bf6c_4e60_aa4c_9af11755eaff.slice/crio-bef2cefe210c3173ceeb235d3b8ddd9426444f5f3bbea5ff8385bcdf7101ddf8 WatchSource:0}: Error finding container bef2cefe210c3173ceeb235d3b8ddd9426444f5f3bbea5ff8385bcdf7101ddf8: Status 404 returned error can't find the container with id bef2cefe210c3173ceeb235d3b8ddd9426444f5f3bbea5ff8385bcdf7101ddf8
Sep 04 23:21:12.593774 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:12.593716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" event={"ID":"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff","Type":"ContainerStarted","Data":"595c32e202b3f02122addb87d015982fc3568e212614930318c8a83eed22a25e"}
Sep 04 23:21:12.594188 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:12.593785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" event={"ID":"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff","Type":"ContainerStarted","Data":"bef2cefe210c3173ceeb235d3b8ddd9426444f5f3bbea5ff8385bcdf7101ddf8"}
Sep 04 23:21:12.610063 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:12.610002    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" podStartSLOduration=1.6099847870000001 podStartE2EDuration="1.609984787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:21:12.608808479 +0000 UTC m=+2156.779883090" watchObservedRunningTime="2026-09-04 23:21:12.609984787 +0000 UTC m=+2156.781059398"
Sep 04 23:21:14.786680 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:14.786639    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:16.719863 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.719829    2577 scope.go:117] "RemoveContainer" containerID="d76b89624f782b679eb1c7cce95d94f1203d2aee5465dbe7709d8e4912db233b"
Sep 04 23:21:16.727644 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.727621    2577 scope.go:117] "RemoveContainer" containerID="ab0d9b962c7fc4e930beb3b94c3fbaf49cd5c8e7337d89555ef1407400587f66"
Sep 04 23:21:16.735207 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.735186    2577 scope.go:117] "RemoveContainer" containerID="e3911f70ef868716cda4b4b57d2f6eb3b63778398f7270518f6b1af8294ef896"
Sep 04 23:21:16.790387 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.790358    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:21:16.790693 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.790676    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:16.790778 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.790697    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6903d24-9481-44bb-9e61-5019dd39615d" containerName="istio-proxy"
Sep 04 23:21:16.795159 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.795140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.800093 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.800069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-24815903-kserve-self-signed-certs\""
Sep 04 23:21:16.805407 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.805386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:21:16.921969 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.921924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.921969 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.921969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.922206 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.921984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.922206 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.922005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.922206 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.922034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj2s\" (UniqueName: \"kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:16.922206 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:16.922077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023351 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj2s\" (UniqueName: \"kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023637 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023838 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023838 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.023936 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.023863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.025707 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.025689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.026000 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.025981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.031495 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.031473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj2s\" (UniqueName: \"kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s\") pod \"e2e-same-id-b-24815903-kserve-7548496f87-44frt\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") " pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.103894 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.103853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:17.372183 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.372158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:21:17.374783 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:21:17.374753    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb005878a_06bf_4d63_8014_2fc257bc1915.slice/crio-b5e12db8a4d818b08720cc3ece1a7da537c261e7296b0f6c1cfba64af45f7897 WatchSource:0}: Error finding container b5e12db8a4d818b08720cc3ece1a7da537c261e7296b0f6c1cfba64af45f7897: Status 404 returned error can't find the container with id b5e12db8a4d818b08720cc3ece1a7da537c261e7296b0f6c1cfba64af45f7897
Sep 04 23:21:17.615931 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.615891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" event={"ID":"b005878a-06bf-4d63-8014-2fc257bc1915","Type":"ContainerStarted","Data":"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"}
Sep 04 23:21:17.616111 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.615941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" event={"ID":"b005878a-06bf-4d63-8014-2fc257bc1915","Type":"ContainerStarted","Data":"b5e12db8a4d818b08720cc3ece1a7da537c261e7296b0f6c1cfba64af45f7897"}
Sep 04 23:21:17.632712 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:17.632615    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" podStartSLOduration=1.63259675 podStartE2EDuration="1.63259675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:21:17.63144013 +0000 UTC m=+2161.802514757" watchObservedRunningTime="2026-09-04 23:21:17.63259675 +0000 UTC m=+2161.803671362"
Sep 04 23:21:21.392750 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:21.392677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:21.392750 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:21.392759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:21.405606 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:21.405572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:21.645543 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:21.645461    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:24.892976 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:24.892941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:27.104285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:27.104229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:27.104285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:27.104302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:27.117217 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:27.117188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:27.670956 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:27.670926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:42.575238 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:42.575197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:44.482250 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:44.482212    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:21:44.503520 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:44.503489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:53.285589 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:53.285555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:21:57.594666 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.594621    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" podUID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" containerName="main" containerID="cri-o://595c32e202b3f02122addb87d015982fc3568e212614930318c8a83eed22a25e" gracePeriod=45
Sep 04 23:21:57.789140 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.789111    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" containerID="595c32e202b3f02122addb87d015982fc3568e212614930318c8a83eed22a25e" exitCode=0
Sep 04 23:21:57.789290 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.789180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" event={"ID":"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff","Type":"ContainerDied","Data":"595c32e202b3f02122addb87d015982fc3568e212614930318c8a83eed22a25e"}
Sep 04 23:21:57.841618 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.841593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:57.993955 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.993864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjwhm\" (UniqueName: \"kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.993960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994130 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm\") pod \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\" (UID: \"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff\") "
Sep 04 23:21:57.994429 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:57.994429 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home" (OuterVolumeSpecName: "home") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:57.994429 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994347    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:57.994556 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994440    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:57.994556 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994459    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:57.994556 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.994476    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:57.996037 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.996010    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm" (OuterVolumeSpecName: "kube-api-access-vjwhm") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "kube-api-access-vjwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:21:57.996156 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.996102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm" (OuterVolumeSpecName: "dshm") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:57.996156 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:57.996140    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" (UID: "1d0aaa25-bf6c-4e60-aa4c-9af11755eaff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:21:58.095583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.095523    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwhm\" (UniqueName: \"kubernetes.io/projected/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-kube-api-access-vjwhm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:58.095583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.095572    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:58.095583 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.095583    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:21:58.794478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.794380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv" event={"ID":"1d0aaa25-bf6c-4e60-aa4c-9af11755eaff","Type":"ContainerDied","Data":"bef2cefe210c3173ceeb235d3b8ddd9426444f5f3bbea5ff8385bcdf7101ddf8"}
Sep 04 23:21:58.794478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.794398    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"
Sep 04 23:21:58.794478 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.794436    2577 scope.go:117] "RemoveContainer" containerID="595c32e202b3f02122addb87d015982fc3568e212614930318c8a83eed22a25e"
Sep 04 23:21:58.815954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.815926    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:58.821205 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:58.821180    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-24815903-kserve-69d7d9ffdc-4wlbv"]
Sep 04 23:21:59.501557 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.501519    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" podUID="b005878a-06bf-4d63-8014-2fc257bc1915" containerName="main" containerID="cri-o://51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3" gracePeriod=45
Sep 04 23:21:59.759697 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.759634    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:59.799603 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.799571    2577 generic.go:356] "Generic (PLEG): container finished" podID="b005878a-06bf-4d63-8014-2fc257bc1915" containerID="51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3" exitCode=0
Sep 04 23:21:59.800057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.799617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"
Sep 04 23:21:59.800057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.799657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" event={"ID":"b005878a-06bf-4d63-8014-2fc257bc1915","Type":"ContainerDied","Data":"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"}
Sep 04 23:21:59.800057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.799717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt" event={"ID":"b005878a-06bf-4d63-8014-2fc257bc1915","Type":"ContainerDied","Data":"b5e12db8a4d818b08720cc3ece1a7da537c261e7296b0f6c1cfba64af45f7897"}
Sep 04 23:21:59.800057 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.799758    2577 scope.go:117] "RemoveContainer" containerID="51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"
Sep 04 23:21:59.807678 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.807663    2577 scope.go:117] "RemoveContainer" containerID="51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"
Sep 04 23:21:59.807958 ip-10-0-134-172 kubenswrapper[2577]: E0904 23:21:59.807939    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3\": container with ID starting with 51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3 not found: ID does not exist" containerID="51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"
Sep 04 23:21:59.808040 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.807966    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3"} err="failed to get container status \"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3\": rpc error: code = NotFound desc = could not find container \"51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3\": container with ID starting with 51498bdb72500e5d59868c4b167a5992c26f91746072be773c53b9e3d0aa21e3 not found: ID does not exist"
Sep 04 23:21:59.909946 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.909912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvj2s\" (UniqueName: \"kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.909999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910066    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910137 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir\") pod \"b005878a-06bf-4d63-8014-2fc257bc1915\" (UID: \"b005878a-06bf-4d63-8014-2fc257bc1915\") "
Sep 04 23:21:59.910409 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache" (OuterVolumeSpecName: "model-cache") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:59.910409 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910352    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home" (OuterVolumeSpecName: "home") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:59.910409 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.910390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:59.912194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.912172    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm" (OuterVolumeSpecName: "dshm") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 23:21:59.912194 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.912184    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 23:21:59.912334 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:21:59.912290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s" (OuterVolumeSpecName: "kube-api-access-rvj2s") pod "b005878a-06bf-4d63-8014-2fc257bc1915" (UID: "b005878a-06bf-4d63-8014-2fc257bc1915"). InnerVolumeSpecName "kube-api-access-rvj2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj2s\" (UniqueName: \"kubernetes.io/projected/b005878a-06bf-4d63-8014-2fc257bc1915-kube-api-access-rvj2s\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011089    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-dshm\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011103    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-model-cache\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011113    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b005878a-06bf-4d63-8014-2fc257bc1915-tls-certs\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011124    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-home\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.011157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.011134    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b005878a-06bf-4d63-8014-2fc257bc1915-tmp-dir\") on node \"ip-10-0-134-172.ec2.internal\" DevicePath \"\""
Sep 04 23:22:00.121022 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.120986    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:22:00.127157 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.127130    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-24815903-kserve-7548496f87-44frt"]
Sep 04 23:22:00.324923 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.324844    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" path="/var/lib/kubelet/pods/1d0aaa25-bf6c-4e60-aa4c-9af11755eaff/volumes"
Sep 04 23:22:00.325249 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:00.325236    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b005878a-06bf-4d63-8014-2fc257bc1915" path="/var/lib/kubelet/pods/b005878a-06bf-4d63-8014-2fc257bc1915/volumes"
Sep 04 23:22:01.986719 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:01.986681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:22:11.189630 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:11.189594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:22:19.394625 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:19.394588    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:22:27.192316 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:27.192277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:22:33.693068 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:33.693032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cwbt6"]
Sep 04 23:22:35.373978 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:35.373940    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6784dc565f-mq2pt_e099cb13-85be-4894-8f7f-2d729efb4fcf/authorino/0.log"
Sep 04 23:22:42.279709 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:42.279632    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759c5d6957-xf8ps_fd98bc56-04c4-42ed-ba22-140583a8f06c/copy-manifests/0.log"
Sep 04 23:22:42.287672 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:42.287646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759c5d6957-xf8ps_fd98bc56-04c4-42ed-ba22-140583a8f06c/manager/0.log"
Sep 04 23:22:43.997946 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:43.997920    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6784dc565f-mq2pt_e099cb13-85be-4894-8f7f-2d729efb4fcf/authorino/0.log"
Sep 04 23:22:44.397265 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:44.397236    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-n9rx4_de8bf063-0a96-49e6-a812-665f62ce36c6/kuadrant-console-plugin/0.log"
Sep 04 23:22:44.683703 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:44.683619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cwbt6_7778dbc0-aa1d-4071-bcb1-baec439b937a/limitador/0.log"
Sep 04 23:22:45.619132 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:45.619106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c4db4689d-r4bvs_1c7b3810-93fb-4407-b305-b5651a1d78f3/kube-auth-proxy/0.log"
Sep 04 23:22:47.045754 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:47.045714    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fsm7xp_398e4815-caf6-49b2-970b-4eb23ff9d86c/main/0.log"
Sep 04 23:22:47.295364 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:47.295338    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-k6pm8_9dab545d-8d66-4541-a517-1639170f09fc/main/0.log"
Sep 04 23:22:53.745894 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.745859    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vpm2f/must-gather-bqcwh"]
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746167    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" containerName="main"
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746179    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" containerName="main"
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746190    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b005878a-06bf-4d63-8014-2fc257bc1915" containerName="main"
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746195    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b005878a-06bf-4d63-8014-2fc257bc1915" containerName="main"
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746245    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d0aaa25-bf6c-4e60-aa4c-9af11755eaff" containerName="main"
Sep 04 23:22:53.746279 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.746253    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b005878a-06bf-4d63-8014-2fc257bc1915" containerName="main"
Sep 04 23:22:53.748954 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.748934    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.751658 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.751638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vpm2f\"/\"kube-root-ca.crt\""
Sep 04 23:22:53.751780 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.751671    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vpm2f\"/\"openshift-service-ca.crt\""
Sep 04 23:22:53.751833 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.751806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vpm2f\"/\"default-dockercfg-s42c6\""
Sep 04 23:22:53.759414 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.759395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vpm2f/must-gather-bqcwh"]
Sep 04 23:22:53.858429 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.858396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t6b\" (UniqueName: \"kubernetes.io/projected/de6d37de-c436-4e06-96ac-c554de3a1d7d-kube-api-access-s4t6b\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.858626 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.858457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de6d37de-c436-4e06-96ac-c554de3a1d7d-must-gather-output\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.959670 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.959625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t6b\" (UniqueName: \"kubernetes.io/projected/de6d37de-c436-4e06-96ac-c554de3a1d7d-kube-api-access-s4t6b\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.959855 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.959695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de6d37de-c436-4e06-96ac-c554de3a1d7d-must-gather-output\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.960022 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.960005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/de6d37de-c436-4e06-96ac-c554de3a1d7d-must-gather-output\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:53.970462 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:53.970429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t6b\" (UniqueName: \"kubernetes.io/projected/de6d37de-c436-4e06-96ac-c554de3a1d7d-kube-api-access-s4t6b\") pod \"must-gather-bqcwh\" (UID: \"de6d37de-c436-4e06-96ac-c554de3a1d7d\") " pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:54.058383 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:54.058284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vpm2f/must-gather-bqcwh"
Sep 04 23:22:54.326297 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:54.326273    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vpm2f/must-gather-bqcwh"]
Sep 04 23:22:54.328751 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:22:54.328712    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde6d37de_c436_4e06_96ac_c554de3a1d7d.slice/crio-a642059abfd62bc4d737c45b1338518ceb16dc50c568f0d970385e9c37325b72 WatchSource:0}: Error finding container a642059abfd62bc4d737c45b1338518ceb16dc50c568f0d970385e9c37325b72: Status 404 returned error can't find the container with id a642059abfd62bc4d737c45b1338518ceb16dc50c568f0d970385e9c37325b72
Sep 04 23:22:54.330568 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:54.330548    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 23:22:55.026123 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:55.026090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vpm2f/must-gather-bqcwh" event={"ID":"de6d37de-c436-4e06-96ac-c554de3a1d7d","Type":"ContainerStarted","Data":"a642059abfd62bc4d737c45b1338518ceb16dc50c568f0d970385e9c37325b72"}
Sep 04 23:22:56.036117 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.036008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vpm2f/must-gather-bqcwh" event={"ID":"de6d37de-c436-4e06-96ac-c554de3a1d7d","Type":"ContainerStarted","Data":"09baf3a8a6f9868fa48e90db2c18f1dd5336e1a7f1cb68e3ff6578a3cf0e145e"}
Sep 04 23:22:56.036117 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.036078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vpm2f/must-gather-bqcwh" event={"ID":"de6d37de-c436-4e06-96ac-c554de3a1d7d","Type":"ContainerStarted","Data":"985cdb32c5bf4ba21ea3badec88312d60649e4f9abb3ea2c8f1efccbf003cf8a"}
Sep 04 23:22:56.055371 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.055307    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vpm2f/must-gather-bqcwh" podStartSLOduration=2.362698404 podStartE2EDuration="3.055284724s" totalImagesPullingTime="692.58632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 23:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:22:56.052518342 +0000 UTC m=+2260.223592954" watchObservedRunningTime="2026-09-04 23:22:56.055284724 +0000 UTC m=+2260.226359338"
Sep 04 23:22:56.535866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.535837    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c75hg_c1fcc2d0-572d-4bc6-91bc-44a34539f922/global-pull-secret-syncer/0.log"
Sep 04 23:22:56.649605 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.649573    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxmgs_7cef06c4-9dc5-4510-96de-855b61f090fb/konnectivity-agent/0.log"
Sep 04 23:22:56.718106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:22:56.718070    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-172.ec2.internal_e90d483920ce499f7a5df446cbd8da0f/haproxy/0.log"
Sep 04 23:23:01.748916 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:01.748839    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6784dc565f-mq2pt_e099cb13-85be-4894-8f7f-2d729efb4fcf/authorino/0.log"
Sep 04 23:23:01.826230 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:01.826198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-n9rx4_de8bf063-0a96-49e6-a812-665f62ce36c6/kuadrant-console-plugin/0.log"
Sep 04 23:23:01.908076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:01.908043    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cwbt6_7778dbc0-aa1d-4071-bcb1-baec439b937a/limitador/0.log"
Sep 04 23:23:03.895761 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:03.895524    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk8sd_18a578e3-edea-4e3d-ac1e-cd4dfb69bb39/node-exporter/0.log"
Sep 04 23:23:03.932306 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:03.932279    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk8sd_18a578e3-edea-4e3d-ac1e-cd4dfb69bb39/kube-rbac-proxy/0.log"
Sep 04 23:23:03.963352 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:03.963319    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk8sd_18a578e3-edea-4e3d-ac1e-cd4dfb69bb39/init-textfile/0.log"
Sep 04 23:23:04.377541 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:04.377478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797975fdf9-7lnv6_79d2fdb6-f33f-435e-93aa-81785774bdd8/telemeter-client/0.log"
Sep 04 23:23:04.398709 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:04.398634    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797975fdf9-7lnv6_79d2fdb6-f33f-435e-93aa-81785774bdd8/reload/0.log"
Sep 04 23:23:04.420379 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:04.420351    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-797975fdf9-7lnv6_79d2fdb6-f33f-435e-93aa-81785774bdd8/kube-rbac-proxy/0.log"
Sep 04 23:23:08.124968 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.124935    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"]
Sep 04 23:23:08.132075 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.132042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.138866 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.138834    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"]
Sep 04 23:23:08.206760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.205725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-sys\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.206760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.205827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-proc\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.206760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.205868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7sh\" (UniqueName: \"kubernetes.io/projected/34dd02bc-48c5-4b5e-a55a-84859a91c93c-kube-api-access-zr7sh\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.206760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.205921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-lib-modules\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.206760 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.205952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-podres\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.307172 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-sys\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.307454 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-proc\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.307601 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7sh\" (UniqueName: \"kubernetes.io/projected/34dd02bc-48c5-4b5e-a55a-84859a91c93c-kube-api-access-zr7sh\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.307762 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-lib-modules\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.307922 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-podres\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.308065 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-proc\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.308196 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.307334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-sys\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.308305 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.308052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-podres\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.308412 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.308139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34dd02bc-48c5-4b5e-a55a-84859a91c93c-lib-modules\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.316041 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.316019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7sh\" (UniqueName: \"kubernetes.io/projected/34dd02bc-48c5-4b5e-a55a-84859a91c93c-kube-api-access-zr7sh\") pod \"perf-node-gather-daemonset-cvw6d\" (UID: \"34dd02bc-48c5-4b5e-a55a-84859a91c93c\") " pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.450847 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.450762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:08.696106 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.696074    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pmhs_217286bb-2071-42aa-b834-bdf9a1b3ad85/dns/0.log"
Sep 04 23:23:08.723604 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.723528    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pmhs_217286bb-2071-42aa-b834-bdf9a1b3ad85/kube-rbac-proxy/0.log"
Sep 04 23:23:08.756038 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.756014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"]
Sep 04 23:23:08.758651 ip-10-0-134-172 kubenswrapper[2577]: W0904 23:23:08.758620    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod34dd02bc_48c5_4b5e_a55a_84859a91c93c.slice/crio-c3ea0752240b8d32fa082a4a1741128b7f050de36e727539b5b906bce32d73f3 WatchSource:0}: Error finding container c3ea0752240b8d32fa082a4a1741128b7f050de36e727539b5b906bce32d73f3: Status 404 returned error can't find the container with id c3ea0752240b8d32fa082a4a1741128b7f050de36e727539b5b906bce32d73f3
Sep 04 23:23:08.877329 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:08.877298    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nhj5d_de5fdc91-819b-4568-bb01-54547f4735cd/dns-node-resolver/0.log"
Sep 04 23:23:09.136551 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.136510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d" event={"ID":"34dd02bc-48c5-4b5e-a55a-84859a91c93c","Type":"ContainerStarted","Data":"ea4e1840ca1890365d05f96fa5466589fda605cfa2d6271aa28be58f07186d8d"}
Sep 04 23:23:09.137034 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.136579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d" event={"ID":"34dd02bc-48c5-4b5e-a55a-84859a91c93c","Type":"ContainerStarted","Data":"c3ea0752240b8d32fa082a4a1741128b7f050de36e727539b5b906bce32d73f3"}
Sep 04 23:23:09.137034 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.136630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:09.154111 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.154048    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d" podStartSLOduration=1.154018304 podStartE2EDuration="1.154018304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 23:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 23:23:09.153005512 +0000 UTC m=+2273.324080130" watchObservedRunningTime="2026-09-04 23:23:09.154018304 +0000 UTC m=+2273.325092911"
Sep 04 23:23:09.373489 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.373456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c9975d4d-dgwj2_50fcc1b1-f4ef-4b49-a24d-6b18a3318353/registry/0.log"
Sep 04 23:23:09.384394 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.384365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85c9975d4d-dgwj2_50fcc1b1-f4ef-4b49-a24d-6b18a3318353/registry/1.log"
Sep 04 23:23:09.429754 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:09.429661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m424m_4ee6f1db-453b-4985-aa29-774fd08ba876/node-ca/0.log"
Sep 04 23:23:10.618300 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:10.618269    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c4db4689d-r4bvs_1c7b3810-93fb-4407-b305-b5651a1d78f3/kube-auth-proxy/0.log"
Sep 04 23:23:11.581035 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:11.580984    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mjcjb_b0618c88-28ff-4047-b6a4-36a3088c7e49/serve-healthcheck-canary/0.log"
Sep 04 23:23:12.139363 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:12.139330    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shfj2_0a175609-ce32-450b-9ac1-7760ff1956a8/kube-rbac-proxy/0.log"
Sep 04 23:23:12.158801 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:12.158770    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shfj2_0a175609-ce32-450b-9ac1-7760ff1956a8/exporter/0.log"
Sep 04 23:23:12.176636 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:12.176604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-shfj2_0a175609-ce32-450b-9ac1-7760ff1956a8/extractor/0.log"
Sep 04 23:23:14.730366 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:14.730333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759c5d6957-xf8ps_fd98bc56-04c4-42ed-ba22-140583a8f06c/manager/0.log"
Sep 04 23:23:14.749239 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:14.749202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-759c5d6957-xf8ps_fd98bc56-04c4-42ed-ba22-140583a8f06c/copy-manifests/0.log"
Sep 04 23:23:15.151248 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:15.151220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vpm2f/perf-node-gather-daemonset-cvw6d"
Sep 04 23:23:15.877285 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:15.877254    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c6bd7c7b8-zcgdg_3d0628fa-a12f-4748-9f29-fa9052233c33/manager/0.log"
Sep 04 23:23:17.156747 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:17.156698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c4db4689d-r4bvs_1c7b3810-93fb-4407-b305-b5651a1d78f3/kube-auth-proxy/0.log"
Sep 04 23:23:19.135865 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:19.135829    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c4db4689d-r4bvs_1c7b3810-93fb-4407-b305-b5651a1d78f3/kube-auth-proxy/0.log"
Sep 04 23:23:22.644625 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.644597    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/kube-multus-additional-cni-plugins/0.log"
Sep 04 23:23:22.661973 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.661944    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/egress-router-binary-copy/0.log"
Sep 04 23:23:22.679158 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.679130    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/cni-plugins/0.log"
Sep 04 23:23:22.703392 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.703369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/bond-cni-plugin/0.log"
Sep 04 23:23:22.724634 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.724606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/routeoverride-cni/0.log"
Sep 04 23:23:22.746852 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.746825    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/whereabouts-cni-bincopy/0.log"
Sep 04 23:23:22.766705 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.766674    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-m4fgs_959228e2-e29a-422b-af73-02a7fa386332/whereabouts-cni/0.log"
Sep 04 23:23:22.978916 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:22.978824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qhmhp_7ca99630-4b48-417c-aff0-efe8419fdb18/kube-multus/0.log"
Sep 04 23:23:23.105142 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.105110    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mcpvb_0333a425-3e65-479b-a1b6-2419b0d01dbe/network-metrics-daemon/0.log"
Sep 04 23:23:23.120884 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.120856    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mcpvb_0333a425-3e65-479b-a1b6-2419b0d01dbe/kube-rbac-proxy/0.log"
Sep 04 23:23:23.887727 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.887694    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/ovn-controller/0.log"
Sep 04 23:23:23.901380 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.901354    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/ovn-acl-logging/0.log"
Sep 04 23:23:23.922098 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.922072    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/kube-rbac-proxy-node/0.log"
Sep 04 23:23:23.943927 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.943892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/kube-rbac-proxy-ovn-metrics/0.log"
Sep 04 23:23:23.957776 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.957754    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/northd/0.log"
Sep 04 23:23:23.975988 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.975963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/nbdb/0.log"
Sep 04 23:23:23.993076 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:23.993035    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/sbdb/0.log"
Sep 04 23:23:24.055795 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:24.055766    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p66rd_21916f8f-fde8-475e-bbfc-c443fcccbe80/ovnkube-controller/0.log"
Sep 04 23:23:25.574961 ip-10-0-134-172 kubenswrapper[2577]: I0904 23:23:25.574930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-nh76j_d5424a3b-48b2-42e4-be05-e476ab369cee/network-check-target-container/0.log"