Sep 08 18:21:37.143337 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 18:21:37.143479 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 18:21:37.143520 ip-10-0-142-80 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 18:21:37.143733 ip-10-0-142-80 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 18:21:47.282423 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 18:21:47.282446 ip-10-0-142-80 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ee907dbf6a664b5baf00f8e17138a5ab --
Sep 08 18:23:44.070854 ip-10-0-142-80 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 18:23:44.493609 ip-10-0-142-80 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:44.493609 ip-10-0-142-80 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 18:23:44.493609 ip-10-0-142-80 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:44.493609 ip-10-0-142-80 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 18:23:44.493609 ip-10-0-142-80 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 18:23:44.495836 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.495743    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 18:23:44.504349 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504327    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.504349 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504345    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.504349 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504349    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.504349 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504354    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504358    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504361    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504364    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504367    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504378    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504382    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504385    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504388    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504390    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504395    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504399    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504404    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504407    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504410    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504414    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504418    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504421    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504423    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.504507 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504427    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504429    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504432    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504434    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504437    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504439    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504443    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504445    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504448    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504451    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504453    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504456    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504458    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504461    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504464    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504467    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504470    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504472    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504475    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504477    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.504976 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504480    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504483    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504486    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504488    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504499    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504502    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504505    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504508    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504510    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504513    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504516    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504518    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504521    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504523    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504526    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504528    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504531    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504535    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504537    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504540    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.505455 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504542    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504559    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504562    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504564    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504567    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504570    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504572    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504576    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504586    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504589    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504591    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504594    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504598    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504602    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504604    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504607    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504610    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504618    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504621    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504624    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.505959 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504626    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504629    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504632    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504634    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504637    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504639    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504642    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504644    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504646    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504649    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504652    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504654    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504656    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504659    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504661    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504664    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.504666    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505399    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505405    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505408    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.506448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505411    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505414    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505417    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505420    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505422    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505425    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505427    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505430    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505433    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505435    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505440    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505443    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505445    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505448    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505451    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505453    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505455    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505458    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505461    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505465    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.506941 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505469    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505472    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505475    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505478    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505481    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505483    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505486    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505489    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505491    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505495    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505497    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505500    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505503    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505505    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505508    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505511    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505513    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505516    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505518    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505521    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.507440 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505524    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505527    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505530    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505532    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505535    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505537    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505540    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505542    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505561    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505564    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505566    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505569    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505571    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505574    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505576    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505579    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505582    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505584    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505587    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505589    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.507943 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505592    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505595    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505597    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505602    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505606    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505608    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505611    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505614    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505616    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505619    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505621    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505624    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505626    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505630    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505632    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505635    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505638    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505641    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505643    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.508454 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505646    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505648    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505651    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505653    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505656    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505658    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505661    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505663    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505666    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505668    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505671    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505674    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505676    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505680    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505684    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505687    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.505690    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506082    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506087    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506090    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.508938 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506093    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506096    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506098    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506101    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506104    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506106    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506109    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506112    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506114    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506117    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506119    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506122    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506126    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506129    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506132    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506134    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506137    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506139    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506142    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506144    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.509486 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506147    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506149    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506152    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506155    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506157    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506159    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506162    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506165    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506170    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506173    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506176    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506178    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506181    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506184    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506186    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506189    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506191    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506194    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506198    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506201    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.510076 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506204    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506207    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506210    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506213    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506216    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506218    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506221    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506225    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506228    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506230    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506233    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506235    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506238    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506241    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506243    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506247    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506249    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506252    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506255    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506258    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.510615 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506264    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506266    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506269    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506272    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506275    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506278    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506280    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506283    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506285    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506288    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506291    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506293    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506295    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506298    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506300    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506303    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506306    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506308    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506310    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.511097 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506313    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506318    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506321    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506324    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506327    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506329    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506332    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506334    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506337    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506339    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506342    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506344    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506347    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506351    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506353    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506356    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.506358    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506435    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506470    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 18:23:44.511565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506477    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506482    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506486    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506489    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506494    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506499    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506502    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506505    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506508    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506511    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506515    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506518    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506521    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506524    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506527    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506529    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506533    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506536    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506539    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506542    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506559    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506563    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506566    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 18:23:44.512067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506569    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506573    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506576    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506580    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506583    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506586    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506589    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506593    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506596    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506599    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506602    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506605    2582 flags.go:64] FLAG: --enable-server="true"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506608    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506612    2582 flags.go:64] FLAG: --event-burst="100"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506615    2582 flags.go:64] FLAG: --event-qps="50"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506618    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506621    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506624    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506628    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506631    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506634    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506637    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506640    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506643    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506645    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 18:23:44.512637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506648    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506651    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506654    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506657    2582 flags.go:64] FLAG: --feature-gates=""
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506660    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506663    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506666    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506669    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506672    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506675    2582 flags.go:64] FLAG: --help="false"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506679    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506682    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506685    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506688    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506691    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506694    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506697    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506700    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506702    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506705    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506708    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506711    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506714    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506717    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 18:23:44.513252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506719    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506722    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506725    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506728    2582 flags.go:64] FLAG: --lock-file=""
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506731    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506734    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506739    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506744    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506747    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506750    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506753    2582 flags.go:64] FLAG: --logging-format="text"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506755    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506759    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506761    2582 flags.go:64] FLAG: --manifest-url=""
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506764    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506768    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506771    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506776    2582 flags.go:64] FLAG: --max-pods="110"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506780    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506783    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506786    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506788    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506791    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506794    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506797    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 18:23:44.514185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506805    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506808    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506811    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506814    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506817    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506823    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506826    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506829    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506832    2582 flags.go:64] FLAG: --port="10250"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506834    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506837    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ea42f6f6e6c32a83"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506840    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506844    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506848    2582 flags.go:64] FLAG: --register-node="true"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506850    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506854    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506857    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506859    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506862    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506866    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506869    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506872    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506874    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506877    2582 flags.go:64] FLAG: --runonce="false"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506880    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 18:23:44.514988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506883    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506886    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506890    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506892    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506895    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506898    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506901    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506904    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506907    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506909    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506912    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506915    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506918    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506921    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506926    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506928    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506931    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506935    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506938    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506941    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506945    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506948    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506951    2582 flags.go:64] FLAG: --v="2"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506955    2582 flags.go:64] FLAG: --version="false"
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506960    2582 flags.go:64] FLAG: --vmodule=""
Sep 08 18:23:44.515617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506968    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.506972    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507103    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507108    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507111    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507114    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507116    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507119    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507123    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507125    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507128    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507130    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507133    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507136    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507138    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507141    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507143    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507146    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507149    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507151    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.516270 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507153    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507156    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507158    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507161    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507163    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507166    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507168    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507172    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507175    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507179    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507181    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507185    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507188    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507191    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507194    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507196    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507200    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507203    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507206    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507209    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.516796 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507214    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507217    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507220    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507222    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507225    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507227    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507230    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507232    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507234    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507237    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507240    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507242    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507245    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507247    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507250    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507253    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507255    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507258    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507260    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507264    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.517286 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507267    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507269    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507272    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507274    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507277    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507280    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507282    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507284    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507287    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507289    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507291    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507294    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507298    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507300    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507303    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507306    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507308    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507311    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507313    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507316    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.517795 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507319    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507321    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507323    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507326    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507328    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507330    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507333    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507335    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507338    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507340    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507343    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507347    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507349    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507352    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507355    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507357    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507361    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507365    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507368    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507370    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.518289 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.507373    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.507378    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.516203    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.516219    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.517017    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.517025    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517084    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517089    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517093    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517097    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517100    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517102    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517106    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517109    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.518791 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517112    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517117    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517121    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517123    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517126    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517129    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517132    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517134    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517137    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517140    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517142    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517145    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517148    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517151    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517155    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517159    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517162    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517165    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517168    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.519144 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517171    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517174    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517177    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517180    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517183    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517185    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517188    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517191    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517193    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517196    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517198    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517201    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517203    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517206    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517209    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517212    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517214    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517217    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517219    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517223    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.519667 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517226    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517229    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517233    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517236    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517239    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517241    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517243    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517246    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517248    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517251    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517253    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517256    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517258    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517261    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517265    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517267    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517270    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517273    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517276    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.520156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517278    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517281    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517283    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517286    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517289    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517292    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517294    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517297    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517300    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517302    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517305    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517307    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517310    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517314    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517316    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517319    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517321    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517324    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517326    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.520648 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517329    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517332    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517334    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517336    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517339    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517342    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517344    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517346    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517349    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517352    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517355    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517357    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517360    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517362    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.521109 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.517368    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517485    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517491    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517494    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517497    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517502    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517505    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517507    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517510    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517512    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517515    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517517    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517520    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517524    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517527    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517530    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517532    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517535    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517537    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 18:23:44.521453 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517539    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517542    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517560    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517564    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517566    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517568    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517571    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517574    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517576    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517579    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517582    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517584    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517587    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517589    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517593    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517595    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517598    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517600    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517603    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517605    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 18:23:44.521928 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517608    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517610    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517613    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517615    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517618    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517620    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517624    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517629    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517632    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517635    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517638    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517641    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517644    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517646    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517649    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517651    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517654    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517657    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517659    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517662    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 18:23:44.522409 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517665    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517668    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517670    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517673    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517676    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517678    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517681    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517683    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517686    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517688    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517691    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517694    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517696    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517699    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517701    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517703    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517706    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517708    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517711    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 18:23:44.522916 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517713    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517716    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517718    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517721    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517723    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517726    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517729    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517733    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517735    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517738    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517741    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517744    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517747    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517750    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517753    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517756    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517759    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517762    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517764    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 18:23:44.523365 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517767    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517769    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:44.517772    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.517777    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.517891    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.519818    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.520994    2582 server.go:1019] "Starting client certificate rotation"
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.521094    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.521135    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 18:23:44.523842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.523516    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:23:44.524733 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.524715    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 18:23:44.535531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.535513    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 08 18:23:44.539761 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.539746    2582 log.go:25] "Validated CRI v1 image API"
Sep 08 18:23:44.540733 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.540708    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 18:23:44.546077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.546043    2582 fs.go:135] Filesystem UUIDs: map[2b9cdc4d-2c71-49eb-bfac-e1ca5e58087c:/dev/nvme0n1p4 33a4715c-86e7-4c7f-a17b-ee6e093bc876:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 08 18:23:44.546181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.546083    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 18:23:44.551026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.551003    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:23:44.551887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.551780    2582 manager.go:216] Machine: {Timestamp:2026-09-08 18:23:44.550671995 +0000 UTC m=+0.358423125 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3047005 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f6eed45a7b5f744a8fbd6938b60a8 SystemUUID:ec2f6eed-45a7-b5f7-44a8-fbd6938b60a8 BootID:ee907dbf-6a66-4b5b-af00-f8e17138a5ab Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:3a:9c:b0:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:3a:9c:b0:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:67:10:95:d9:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 18:23:44.551887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.551886    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 18:23:44.551993 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.551971    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 18:23:44.555555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.555523    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 18:23:44.555699 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.555559    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 18:23:44.556451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.556441    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 18:23:44.556485 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.556453    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 18:23:44.556485 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.556462    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:23:44.557410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.557400    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 18:23:44.558590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.558581    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 18:23:44.561351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561339    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:23:44.561466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561456    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 18:23:44.561499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561486    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 18:23:44.561499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561495    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 18:23:44.561567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561509    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 18:23:44.561567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561518    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 08 18:23:44.561567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561527    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 18:23:44.561953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561941    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:23:44.561990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.561962    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 18:23:44.564631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.564611    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 18:23:44.566825 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.566811    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 18:23:44.566877 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.566842    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 18:23:44.567513 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567501    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 18:23:44.567568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567521    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 18:23:44.567568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567527    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 18:23:44.567568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567532    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 18:23:44.567568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567541    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 18:23:44.567568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567563    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 18:23:44.567794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567786    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 18:23:44.567794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567794    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 18:23:44.567849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567801    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 18:23:44.567849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567806    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 18:23:44.567849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567815    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 18:23:44.567849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.567822    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 18:23:44.569192 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.569181    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 18:23:44.569192 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.569192    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 18:23:44.572591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.572573    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 18:23:44.572681 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.572595    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 18:23:44.572681 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.572644    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 18:23:44.573223 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.573211    2582 server.go:1268] "Started kubelet"
Sep 08 18:23:44.573298 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.573272    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 18:23:44.573819 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.573763    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 18:23:44.573909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.573850    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 18:23:44.574534 ip-10-0-142-80 systemd[1]: Started Kubernetes Kubelet.
Sep 08 18:23:44.575021 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.574893    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 18:23:44.575628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.575616    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 18:23:44.577853 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.577831    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 18:23:44.578736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.578719    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 18:23:44.579362 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.579337    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 18:23:44.579479 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.579460    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.579999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.579976    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 18:23:44.580653 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.580538    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 18:23:44.580653 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.580649    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 18:23:44.580851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.580830    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 18:23:44.580851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.580843    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 18:23:44.581011 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.580989    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m5zj7"
Sep 08 18:23:44.581696 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.581679    2582 factory.go:55] Registering systemd factory
Sep 08 18:23:44.581782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.581720    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 08 18:23:44.581977 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.581960    2582 factory.go:153] Registering CRI-O factory
Sep 08 18:23:44.582059 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.581980    2582 factory.go:223] Registration of the crio container factory successfully
Sep 08 18:23:44.582281 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.582061    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 18:23:44.582281 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.582080    2582 factory.go:103] Registering Raw factory
Sep 08 18:23:44.582281 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.582109    2582 manager.go:1196] Started watching for new ooms in manager
Sep 08 18:23:44.582494 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.582476    2582 manager.go:349] Starting recovery of all containers
Sep 08 18:23:44.586663 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.586633    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 18:23:44.586840 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.586818    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 18:23:44.588341 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.588321    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m5zj7"
Sep 08 18:23:44.590197 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.586501    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-80.ec2.internal.18d36ba2f148f48f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-80.ec2.internal,UID:ip-10-0-142-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-80.ec2.internal,},FirstTimestamp:2026-09-08 18:23:44.573183119 +0000 UTC m=+0.380934250,LastTimestamp:2026-09-08 18:23:44.573183119 +0000 UTC m=+0.380934250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-80.ec2.internal,}"
Sep 08 18:23:44.596716 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.596699    2582 manager.go:354] Recovery completed
Sep 08 18:23:44.600794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.600779    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.602843 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.602825    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.602911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.602856    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.602911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.602880    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.603618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.603543    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 18:23:44.603618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.603583    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 18:23:44.603618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.603599    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 18:23:44.603735 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.603561    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.604883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.604870    2582 policy_none.go:49] "None policy: Start"
Sep 08 18:23:44.604934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.604890    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 18:23:44.604934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.604904    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 18:23:44.605074 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.604982    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-80.ec2.internal.18d36ba2f30d80e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-80.ec2.internal,UID:ip-10-0-142-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-80.ec2.internal,},FirstTimestamp:2026-09-08 18:23:44.602841321 +0000 UTC m=+0.410592451,LastTimestamp:2026-09-08 18:23:44.602841321 +0000 UTC m=+0.410592451,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-80.ec2.internal,}"
Sep 08 18:23:44.605911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.605900    2582 policy_none.go:47] "Start"
Sep 08 18:23:44.606176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.606163    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.606219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.606189    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.606219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.606200    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.651234 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651216    2582 manager.go:337] "Starting Device Plugin manager"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.651267    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651278    2582 server.go:87] "Starting device plugin registration server"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651637    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651650    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651766    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651846    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.651855    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.652423    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 18:23:44.660212 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.652481    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.734912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.734874    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 18:23:44.736282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.736260    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 18:23:44.736422 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.736314    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 18:23:44.736422 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.736342    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 18:23:44.736422 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.736397    2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 18:23:44.739121 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.739093    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:44.752194 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.752123    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.753043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.753027    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.753098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.753058    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.753098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.753069    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.753098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.753093    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.761988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.761975    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.762039 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.761996    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-80.ec2.internal\": node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.778022 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.777999    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.836489 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.836450    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"]
Sep 08 18:23:44.836656 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.836537    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.843040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.843018    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.843104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.843058    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.843104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.843069    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.844299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.844286    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.844412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.844397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.844467 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.844431    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.845148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845128    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.845242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845159    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.845242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845172    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.845242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845180    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.845242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845218    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.845242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.845229    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.846284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.846271    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.846345 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.846294    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 18:23:44.848537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.848518    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 18:23:44.848624 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.848570    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 18:23:44.848624 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.848586    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 08 18:23:44.876178 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.876159    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-80.ec2.internal\" not found" node="ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.878141 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.878125    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.880766 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.880751    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-80.ec2.internal\" not found" node="ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.882883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.882863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.882941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.882896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.883025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.882915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b7a1699332bcb5df721e142c96ab97c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"9b7a1699332bcb5df721e142c96ab97c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.978705 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:44.978666    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:44.984036 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.984123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.984123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b7a1699332bcb5df721e142c96ab97c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"9b7a1699332bcb5df721e142c96ab97c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.984215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984129    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b7a1699332bcb5df721e142c96ab97c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"9b7a1699332bcb5df721e142c96ab97c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.984215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:44.984215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:44.984140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f2063da704829bef1181fbc5366292c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"7f2063da704829bef1181fbc5366292c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:45.079459 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.079377    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:45.178902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.178863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:45.179965 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.179944    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:45.183628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.183599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:45.280257 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.280214    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:45.380745 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.380660    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 08 18:23:45.433315 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.433295    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:45.480758 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.480730    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:45.493003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.492981    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:23:45.494972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.494958    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 08 18:23:45.507021 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.507004    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 18:23:45.521145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.521121    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 18:23:45.521261 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.521244    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 18:23:45.521310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.521286    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 18:23:45.562428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.562401    2582 apiserver.go:52] "Watching apiserver"
Sep 08 18:23:45.579065 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.579044    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 18:23:45.579123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.579051    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 18:23:45.579459 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.579437    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-9br6q","openshift-network-diagnostics/network-check-target-x7tgt","kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5c2x7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal","openshift-multus/multus-additional-cni-plugins-8r4mg","openshift-network-operator/iptables-alerter-bdp8t","openshift-ovn-kubernetes/ovnkube-node-xlpjx","kube-system/global-pull-secret-syncer-sb25t","kube-system/konnectivity-agent-pbmrz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8","openshift-dns/node-resolver-qwcwx","openshift-image-registry/node-ca-kf9x2","openshift-multus/multus-hwhsw"]
Sep 08 18:23:45.581122 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.581108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.581208 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.581189    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:45.582196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.582180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:45.582266 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.582248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:45.583350 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.583331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.584487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.584472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.585647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.585628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.585726 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.585706    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qsl94\""
Sep 08 18:23:45.585921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.585907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.586217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.586200    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.586713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.586698    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 18:23:45.586891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.586874    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 18:23:45.586982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.586966    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.587035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.586994    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.587082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.587069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7fqs4\""
Sep 08 18:23:45.587665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.587638    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.587665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.587808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 18:23:45.588082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.588145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7d4\" (UniqueName: \"kubernetes.io/projected/623bbd40-a928-4099-9d04-964fa3145559-kube-api-access-6c7d4\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.588192 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:45.588735 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588714    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.588802 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7rvk\""
Sep 08 18:23:45.588993 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.588976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 18:23:45.589097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.589080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.589371 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.589354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.589442 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.589422    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:45.589917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.589901    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.590028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.590013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.590195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.590155    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 18:18:45 +0000 UTC" deadline="2028-05-06 00:44:13.612234123 +0000 UTC"
Sep 08 18:23:45.590248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.590196    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14526h20m28.022041347s"
Sep 08 18:23:45.590397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.590383    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 18:23:45.590482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.590465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.591479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.591460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 18:23:45.591601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.591519    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 18:23:45.591601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.591527    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 18:23:45.591601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.591535    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qbh94\""
Sep 08 18:23:45.592260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.592247    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.592395 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.592368    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 18:23:45.592853 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.592838    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 18:23:45.592933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.592868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jhtvb\""
Sep 08 18:23:45.593270 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.593257    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 18:23:45.593452 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.593440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.594525 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.594510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.594685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.594669    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.594839 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.594824    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 18:23:45.595097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.595082    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.595170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.595110    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6gtv\""
Sep 08 18:23:45.595639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.595622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.595851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.595829    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.596159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.596140    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jx2bc\""
Sep 08 18:23:45.596159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.596153    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.596903 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.596873    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 18:23:45.597067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.597054    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 18:23:45.597116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.597069    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 18:23:45.597157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.597134    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqxjc\""
Sep 08 18:23:45.597972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.597958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 18:23:45.598092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.598078    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zsjpb\""
Sep 08 18:23:45.619434 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.619400    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lskr7"
Sep 08 18:23:45.628068 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.628049    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lskr7"
Sep 08 18:23:45.662531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.662509    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:45.681746 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.681724    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 18:23:45.689184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-conf-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-systemd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-log-socket\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-bin\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjrl\" (UniqueName: \"kubernetes.io/projected/93a88fc2-f535-4304-ae16-cd66d5924739-kube-api-access-qfjrl\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-cnibin\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.689351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-registration-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ebeebd7-59c9-428f-bb26-025899cc0bb8-konnectivity-ca\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689405    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a88fc2-f535-4304-ae16-cd66d5924739-tmp-dir\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-sys\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689464    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-host\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cnibin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689514    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-socket-dir-parent\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-dbus\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.689736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.689609    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7d4\" (UniqueName: \"kubernetes.io/projected/623bbd40-a928-4099-9d04-964fa3145559-kube-api-access-6c7d4\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690184    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-etc-kubernetes\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ebeebd7-59c9-428f-bb26-025899cc0bb8-agent-certs\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-modprobe-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-multus\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-run\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-bin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqc4w\" (UniqueName: \"kubernetes.io/projected/5fe5c088-94d4-4752-95c8-d3d4691547df-kube-api-access-gqc4w\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690711    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6ql\" (UniqueName: \"kubernetes.io/projected/6d8b56b1-e175-437f-bd9d-a0994baf64ad-kube-api-access-2c6ql\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.690841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62dbe9f7-1c35-42c3-be25-65e52c6409c4-serviceca\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-kubernetes\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-etc-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.690970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-conf\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691004    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-tuned\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-system-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-kubelet\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-var-lib-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93a88fc2-f535-4304-ae16-cd66d5924739-hosts-file\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-daemon-config\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-multus-certs\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.691598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-ovn\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgrmd\" (UniqueName: \"kubernetes.io/projected/62dbe9f7-1c35-42c3-be25-65e52c6409c4-kube-api-access-zgrmd\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-var-lib-kubelet\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-tmp\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.691965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62dbe9f7-1c35-42c3-be25-65e52c6409c4-host\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-systemd\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692037    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-lib-modules\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnwf\" (UniqueName: \"kubernetes.io/projected/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-kube-api-access-7hnwf\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-node-log\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.692206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-script-lib\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.692702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.692702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.692702 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.692382    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:45.692702 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.692459    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:46.192436582 +0000 UTC m=+2.000187701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:45.692702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-device-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.692968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cni-binary-copy\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.692968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692818    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-k8s-cni-cncf-io\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.692968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-netns\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.692968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-kubelet\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.692968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-systemd-units\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.692980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysconfig\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-sys-fs\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-env-overrides\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95jt\" (UniqueName: \"kubernetes.io/projected/1b01fcde-41df-450e-8ee7-f833570fa0f5-kube-api-access-d95jt\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693123    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-socket-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.693202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-hostroot\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-slash\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-kubelet-config\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-system-cni-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztls\" (UniqueName: \"kubernetes.io/projected/6c7acc60-35eb-4357-bc47-5bfed8c77926-kube-api-access-pztls\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkxhj\" (UniqueName: \"kubernetes.io/projected/33c186c4-115e-4335-bb21-814c8d662096-kube-api-access-kkxhj\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.693493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693415    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fe5c088-94d4-4752-95c8-d3d4691547df-iptables-alerter-script\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.693793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fe5c088-94d4-4752-95c8-d3d4691547df-host-slash\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.693793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-netns\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-config\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.693793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.693680    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-netd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.696737 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.696718    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 18:23:45.700089 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.700069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7d4\" (UniqueName: \"kubernetes.io/projected/623bbd40-a928-4099-9d04-964fa3145559-kube-api-access-6c7d4\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:45.701173 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.701156    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:45.701236 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.701177    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:45.701236 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.701187    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:45.701313 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.701236    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:23:46.201222703 +0000 UTC m=+2.008973836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:45.767301 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:45.767273    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7a1699332bcb5df721e142c96ab97c.slice/crio-ca5ed82d7b062dc7753843ea35f148ffc780999b813f82c38c494ed0b29101e7 WatchSource:0}: Error finding container ca5ed82d7b062dc7753843ea35f148ffc780999b813f82c38c494ed0b29101e7: Status 404 returned error can't find the container with id ca5ed82d7b062dc7753843ea35f148ffc780999b813f82c38c494ed0b29101e7
Sep 08 18:23:45.767790 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:45.767772    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2063da704829bef1181fbc5366292c.slice/crio-00ca3865bc7e268a18576ca8ea6888834fbc9c4cb0b6f8b37017c456ec223cbe WatchSource:0}: Error finding container 00ca3865bc7e268a18576ca8ea6888834fbc9c4cb0b6f8b37017c456ec223cbe: Status 404 returned error can't find the container with id 00ca3865bc7e268a18576ca8ea6888834fbc9c4cb0b6f8b37017c456ec223cbe
Sep 08 18:23:45.770525 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.770509    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:23:45.794359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-conf\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.794359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794356    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-tuned\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.794528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.794528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-system-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.794780 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-conf\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.794820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-kubelet\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.794860 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-var-lib-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.794915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93a88fc2-f535-4304-ae16-cd66d5924739-hosts-file\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.794915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-kubelet\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.794915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.794915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794659    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-system-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.794915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-daemon-config\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-var-lib-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-multus-certs\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-ovn\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgrmd\" (UniqueName: \"kubernetes.io/projected/62dbe9f7-1c35-42c3-be25-65e52c6409c4-kube-api-access-zgrmd\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-var-lib-kubelet\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-multus-certs\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-cni-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-ovn\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-tmp\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.794936    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93a88fc2-f535-4304-ae16-cd66d5924739-hosts-file\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-var-lib-kubelet\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62dbe9f7-1c35-42c3-be25-65e52c6409c4-host\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.795153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62dbe9f7-1c35-42c3-be25-65e52c6409c4-host\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-systemd\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-lib-modules\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795187    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnwf\" (UniqueName: \"kubernetes.io/projected/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-kube-api-access-7hnwf\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-systemd\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-node-log\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-lib-modules\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-node-log\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-script-lib\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-daemon-config\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-device-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cni-binary-copy\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-k8s-cni-cncf-io\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-netns\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.795884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-device-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-kubelet\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-systemd-units\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795646    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysconfig\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795686    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysctl-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-sys-fs\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-env-overrides\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95jt\" (UniqueName: \"kubernetes.io/projected/1b01fcde-41df-450e-8ee7-f833570fa0f5-kube-api-access-d95jt\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795726    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-kubelet\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-socket-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-k8s-cni-cncf-io\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-hostroot\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-run-netns\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-slash\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-sys-fs\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.796749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-kubelet-config\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-systemd-units\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-system-cni-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-sysconfig\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztls\" (UniqueName: \"kubernetes.io/projected/6c7acc60-35eb-4357-bc47-5bfed8c77926-kube-api-access-pztls\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.795978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkxhj\" (UniqueName: \"kubernetes.io/projected/33c186c4-115e-4335-bb21-814c8d662096-kube-api-access-kkxhj\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fe5c088-94d4-4752-95c8-d3d4691547df-iptables-alerter-script\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fe5c088-94d4-4752-95c8-d3d4691547df-host-slash\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796065    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-binary-copy\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-netns\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cni-binary-copy\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-config\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-netd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-kubelet-config\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796169    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-conf-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.797632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-script-lib\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-systemd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-socket-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-log-socket\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-log-socket\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-systemd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-bin\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-system-cni-dir\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796264    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-bin\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796403    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-etc-selinux\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796408    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-netns\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796460    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-conf-dir\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796461    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-cni-netd\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-slash\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-env-overrides\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-hostroot\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjrl\" (UniqueName: \"kubernetes.io/projected/93a88fc2-f535-4304-ae16-cd66d5924739-kube-api-access-qfjrl\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.798231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fe5c088-94d4-4752-95c8-d3d4691547df-host-slash\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-cnibin\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c7acc60-35eb-4357-bc47-5bfed8c77926-cnibin\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-registration-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ebeebd7-59c9-428f-bb26-025899cc0bb8-konnectivity-ca\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a88fc2-f535-4304-ae16-cd66d5924739-tmp-dir\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-sys\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-host\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cnibin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-socket-dir-parent\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-dbus\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-etc-kubernetes\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ebeebd7-59c9-428f-bb26-025899cc0bb8-agent-certs\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-modprobe-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.798803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-multus\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796976    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovnkube-config\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a88fc2-f535-4304-ae16-cd66d5924739-tmp-dir\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-run\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-run\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797079    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-sys\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-bin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6c7acc60-35eb-4357-bc47-5bfed8c77926-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqc4w\" (UniqueName: \"kubernetes.io/projected/5fe5c088-94d4-4752-95c8-d3d4691547df-kube-api-access-gqc4w\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797138    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-etc-kubernetes\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.796720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/33c186c4-115e-4335-bb21-814c8d662096-registration-dir\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6ql\" (UniqueName: \"kubernetes.io/projected/6d8b56b1-e175-437f-bd9d-a0994baf64ad-kube-api-access-2c6ql\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797200    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-host\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62dbe9f7-1c35-42c3-be25-65e52c6409c4-serviceca\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-cnibin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-kubernetes\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ebeebd7-59c9-428f-bb26-025899cc0bb8-konnectivity-ca\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-etc-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-bin\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797295    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-kubernetes\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797324    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-multus-socket-dir-parent\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.797392    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-host-var-lib-cni-multus\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-dbus\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-etc-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fe5c088-94d4-4752-95c8-d3d4691547df-iptables-alerter-script\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:45.797476    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:23:46.297459975 +0000 UTC m=+2.105211094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b01fcde-41df-450e-8ee7-f833570fa0f5-run-openvswitch\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-modprobe-d\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-etc-tuned\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d8b56b1-e175-437f-bd9d-a0994baf64ad-tmp\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.797885    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62dbe9f7-1c35-42c3-be25-65e52c6409c4-serviceca\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.799901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.799482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ebeebd7-59c9-428f-bb26-025899cc0bb8-agent-certs\") pod \"konnectivity-agent-pbmrz\" (UID: \"2ebeebd7-59c9-428f-bb26-025899cc0bb8\") " pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:45.800365 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.799625    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b01fcde-41df-450e-8ee7-f833570fa0f5-ovn-node-metrics-cert\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.811016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.806753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgrmd\" (UniqueName: \"kubernetes.io/projected/62dbe9f7-1c35-42c3-be25-65e52c6409c4-kube-api-access-zgrmd\") pod \"node-ca-kf9x2\" (UID: \"62dbe9f7-1c35-42c3-be25-65e52c6409c4\") " pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:45.811016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.809679    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkxhj\" (UniqueName: \"kubernetes.io/projected/33c186c4-115e-4335-bb21-814c8d662096-kube-api-access-kkxhj\") pod \"aws-ebs-csi-driver-node-xbwv8\" (UID: \"33c186c4-115e-4335-bb21-814c8d662096\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:45.811016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.810002    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnwf\" (UniqueName: \"kubernetes.io/projected/3dce7655-4f24-49ff-a2b4-85a44d40bf5f-kube-api-access-7hnwf\") pod \"multus-hwhsw\" (UID: \"3dce7655-4f24-49ff-a2b4-85a44d40bf5f\") " pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:45.811016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.810395    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjrl\" (UniqueName: \"kubernetes.io/projected/93a88fc2-f535-4304-ae16-cd66d5924739-kube-api-access-qfjrl\") pod \"node-resolver-qwcwx\" (UID: \"93a88fc2-f535-4304-ae16-cd66d5924739\") " pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:45.811016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.810539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95jt\" (UniqueName: \"kubernetes.io/projected/1b01fcde-41df-450e-8ee7-f833570fa0f5-kube-api-access-d95jt\") pod \"ovnkube-node-xlpjx\" (UID: \"1b01fcde-41df-450e-8ee7-f833570fa0f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:45.812128 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.812096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6ql\" (UniqueName: \"kubernetes.io/projected/6d8b56b1-e175-437f-bd9d-a0994baf64ad-kube-api-access-2c6ql\") pod \"tuned-5c2x7\" (UID: \"6d8b56b1-e175-437f-bd9d-a0994baf64ad\") " pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:45.812226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.812128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqc4w\" (UniqueName: \"kubernetes.io/projected/5fe5c088-94d4-4752-95c8-d3d4691547df-kube-api-access-gqc4w\") pod \"iptables-alerter-bdp8t\" (UID: \"5fe5c088-94d4-4752-95c8-d3d4691547df\") " pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:45.812316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.812300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztls\" (UniqueName: \"kubernetes.io/projected/6c7acc60-35eb-4357-bc47-5bfed8c77926-kube-api-access-pztls\") pod \"multus-additional-cni-plugins-8r4mg\" (UID: \"6c7acc60-35eb-4357-bc47-5bfed8c77926\") " pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:45.814294 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.814276    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:45.923007 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:45.922917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5c2x7"
Sep 08 18:23:47.288675 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.288651    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8b56b1_e175_437f_bd9d_a0994baf64ad.slice/crio-60b7c207200d335a4348d6cc3368cea7a6c54fbd49f8ee255c409f3419f8afc8 WatchSource:0}: Error finding container 60b7c207200d335a4348d6cc3368cea7a6c54fbd49f8ee255c409f3419f8afc8: Status 404 returned error can't find the container with id 60b7c207200d335a4348d6cc3368cea7a6c54fbd49f8ee255c409f3419f8afc8
Sep 08 18:23:47.299229 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.299207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8r4mg"
Sep 08 18:23:47.306108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.306081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bdp8t"
Sep 08 18:23:47.306448 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.306426    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7acc60_35eb_4357_bc47_5bfed8c77926.slice/crio-92421a225974f1f8fd8be370815227420cc2290150b5a44d0b9ee54c3739065e WatchSource:0}: Error finding container 92421a225974f1f8fd8be370815227420cc2290150b5a44d0b9ee54c3739065e: Status 404 returned error can't find the container with id 92421a225974f1f8fd8be370815227420cc2290150b5a44d0b9ee54c3739065e
Sep 08 18:23:47.313947 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.313923    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe5c088_94d4_4752_95c8_d3d4691547df.slice/crio-4329b9529c2fd2c4375e0e394e4a31e939337e97e1ad31731285561749f2852c WatchSource:0}: Error finding container 4329b9529c2fd2c4375e0e394e4a31e939337e97e1ad31731285561749f2852c: Status 404 returned error can't find the container with id 4329b9529c2fd2c4375e0e394e4a31e939337e97e1ad31731285561749f2852c
Sep 08 18:23:47.329533 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.329508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:23:47.338990 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.338968    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b01fcde_41df_450e_8ee7_f833570fa0f5.slice/crio-9af1848843ecba16bdbc619a91dc0ca1da2a753e0dcfe68b7d0256523e7c4725 WatchSource:0}: Error finding container 9af1848843ecba16bdbc619a91dc0ca1da2a753e0dcfe68b7d0256523e7c4725: Status 404 returned error can't find the container with id 9af1848843ecba16bdbc619a91dc0ca1da2a753e0dcfe68b7d0256523e7c4725
Sep 08 18:23:47.344139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.344117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:23:47.351268 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.351242    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebeebd7_59c9_428f_bb26_025899cc0bb8.slice/crio-bad6d325d8c18e9ad2aca15102de53d35157828b7e795f8e2c22afdc361ef53f WatchSource:0}: Error finding container bad6d325d8c18e9ad2aca15102de53d35157828b7e795f8e2c22afdc361ef53f: Status 404 returned error can't find the container with id bad6d325d8c18e9ad2aca15102de53d35157828b7e795f8e2c22afdc361ef53f
Sep 08 18:23:47.363239 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.363215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8"
Sep 08 18:23:47.371587 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.371562    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c186c4_115e_4335_bb21_814c8d662096.slice/crio-580eaf1bfbbaf01e7664e89ce63482ba9e4c7937843e718abce10f3d22c2b380 WatchSource:0}: Error finding container 580eaf1bfbbaf01e7664e89ce63482ba9e4c7937843e718abce10f3d22c2b380: Status 404 returned error can't find the container with id 580eaf1bfbbaf01e7664e89ce63482ba9e4c7937843e718abce10f3d22c2b380
Sep 08 18:23:47.376752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.376709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qwcwx"
Sep 08 18:23:47.383879 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.383859    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a88fc2_f535_4304_ae16_cd66d5924739.slice/crio-0531b82533fdab22a3e30f498c229752df5bfbda97ab7638e1a2ac513e5c4858 WatchSource:0}: Error finding container 0531b82533fdab22a3e30f498c229752df5bfbda97ab7638e1a2ac513e5c4858: Status 404 returned error can't find the container with id 0531b82533fdab22a3e30f498c229752df5bfbda97ab7638e1a2ac513e5c4858
Sep 08 18:23:47.389871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.389849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kf9x2"
Sep 08 18:23:47.393975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.393959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwhsw"
Sep 08 18:23:47.397556 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.397534    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dbe9f7_1c35_42c3_be25_65e52c6409c4.slice/crio-c2d7b668e34e9a50ee2606c232b5893b6661fb7cfbb1c92bd31f482453cd0686 WatchSource:0}: Error finding container c2d7b668e34e9a50ee2606c232b5893b6661fb7cfbb1c92bd31f482453cd0686: Status 404 returned error can't find the container with id c2d7b668e34e9a50ee2606c232b5893b6661fb7cfbb1c92bd31f482453cd0686
Sep 08 18:23:47.401746 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:23:47.401701    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dce7655_4f24_49ff_a2b4_85a44d40bf5f.slice/crio-41e8dcc4d1ae42f326e70051bfcd8f06b554ff4fbc6298aca63ed4901738b938 WatchSource:0}: Error finding container 41e8dcc4d1ae42f326e70051bfcd8f06b554ff4fbc6298aca63ed4901738b938: Status 404 returned error can't find the container with id 41e8dcc4d1ae42f326e70051bfcd8f06b554ff4fbc6298aca63ed4901738b938
Sep 08 18:23:47.555307 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.555225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:47.555435 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.555329    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:47.555435 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.555379    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:48.555365641 +0000 UTC m=+3.006940750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:47.655852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.655813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:47.656008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.655864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:47.656008 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.655930    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:47.656008 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.655994    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:23:48.655975015 +0000 UTC m=+3.107550130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:47.656214 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.656026    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:47.656214 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.656042    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:47.656214 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.656051    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:47.656214 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:47.656087    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:23:48.656075138 +0000 UTC m=+3.107650266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:47.894939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.894849    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:47.986114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.986040    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:18:46 +0000 UTC" deadline="2028-05-06 00:35:47.951450825 +0000 UTC"
Sep 08 18:23:47.986114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:47.986110    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14526h11m59.96534508s"
Sep 08 18:23:48.113487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.113424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8" event={"ID":"33c186c4-115e-4335-bb21-814c8d662096","Type":"ContainerStarted","Data":"580eaf1bfbbaf01e7664e89ce63482ba9e4c7937843e718abce10f3d22c2b380"}
Sep 08 18:23:48.123433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.123349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pbmrz" event={"ID":"2ebeebd7-59c9-428f-bb26-025899cc0bb8","Type":"ContainerStarted","Data":"bad6d325d8c18e9ad2aca15102de53d35157828b7e795f8e2c22afdc361ef53f"}
Sep 08 18:23:48.143676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.143639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerStarted","Data":"92421a225974f1f8fd8be370815227420cc2290150b5a44d0b9ee54c3739065e"}
Sep 08 18:23:48.161717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.161638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" event={"ID":"9b7a1699332bcb5df721e142c96ab97c","Type":"ContainerStarted","Data":"ca5ed82d7b062dc7753843ea35f148ffc780999b813f82c38c494ed0b29101e7"}
Sep 08 18:23:48.171345 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.171284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwhsw" event={"ID":"3dce7655-4f24-49ff-a2b4-85a44d40bf5f","Type":"ContainerStarted","Data":"41e8dcc4d1ae42f326e70051bfcd8f06b554ff4fbc6298aca63ed4901738b938"}
Sep 08 18:23:48.177167 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.177136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kf9x2" event={"ID":"62dbe9f7-1c35-42c3-be25-65e52c6409c4","Type":"ContainerStarted","Data":"c2d7b668e34e9a50ee2606c232b5893b6661fb7cfbb1c92bd31f482453cd0686"}
Sep 08 18:23:48.187236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.186924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"9af1848843ecba16bdbc619a91dc0ca1da2a753e0dcfe68b7d0256523e7c4725"}
Sep 08 18:23:48.198736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.198691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdp8t" event={"ID":"5fe5c088-94d4-4752-95c8-d3d4691547df","Type":"ContainerStarted","Data":"4329b9529c2fd2c4375e0e394e4a31e939337e97e1ad31731285561749f2852c"}
Sep 08 18:23:48.214136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.214098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5c2x7" event={"ID":"6d8b56b1-e175-437f-bd9d-a0994baf64ad","Type":"ContainerStarted","Data":"60b7c207200d335a4348d6cc3368cea7a6c54fbd49f8ee255c409f3419f8afc8"}
Sep 08 18:23:48.220538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.220505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"7f2063da704829bef1181fbc5366292c","Type":"ContainerStarted","Data":"00ca3865bc7e268a18576ca8ea6888834fbc9c4cb0b6f8b37017c456ec223cbe"}
Sep 08 18:23:48.225806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.225778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwcwx" event={"ID":"93a88fc2-f535-4304-ae16-cd66d5924739","Type":"ContainerStarted","Data":"0531b82533fdab22a3e30f498c229752df5bfbda97ab7638e1a2ac513e5c4858"}
Sep 08 18:23:48.565304 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.565208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:48.565484 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.565417    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:48.565484 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.565481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:50.565464033 +0000 UTC m=+5.017039158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:48.666471 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.666429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:48.666656 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.666486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:48.666749 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666670    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:48.666749 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666688    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:48.666749 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666701    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:48.666902 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666775    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:23:50.666757131 +0000 UTC m=+5.118332245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:48.666902 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666851    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:48.666902 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:48.666884    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:23:50.666872915 +0000 UTC m=+5.118448029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:48.986520 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.986472    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:18:46 +0000 UTC" deadline="2028-06-02 22:34:21.229820176 +0000 UTC"
Sep 08 18:23:48.986520 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:48.986519    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15196h10m32.24330602s"
Sep 08 18:23:49.093644 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:49.093606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:49.093842 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:49.093785    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:49.094243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:49.094217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:49.094369 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:49.094344    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:49.094437 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:49.094422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:49.094524 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:49.094503    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:49.134317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:49.134247    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 18:23:50.583532 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:50.582903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:50.583532 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.583078    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:50.583532 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.583146    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:23:54.583125649 +0000 UTC m=+9.034700765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:50.683894 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:50.683852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:50.684071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:50.683907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:50.684133 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684097    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:50.684133 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684120    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:50.684133 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684134    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:50.684276 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684197    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:23:54.684178741 +0000 UTC m=+9.135753872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:50.684912 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684661    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:50.684912 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:50.684745    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:23:54.684710388 +0000 UTC m=+9.136285521 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:51.093187    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:51.093341    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:51.093430    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:51.093509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:51.093574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:51.094116 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:51.093642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:53.093788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:53.093751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:53.094211 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:53.093901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:53.094257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:53.094235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:53.094335 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:53.094301    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:53.094454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:53.094344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:53.094454 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:53.094415    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:54.616851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:54.616263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:54.616851 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.616453    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:54.616851 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.616513    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:02.616498052 +0000 UTC m=+17.068073185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:23:54.717037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:54.716995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:54.717208 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:54.717093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717281    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717307    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717320    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717388    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:24:02.717369255 +0000 UTC m=+17.168944389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717399    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:54.717468 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:54.717437    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:24:02.717425355 +0000 UTC m=+17.169000464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:23:55.093573 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:55.093541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:55.093573 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:55.093571    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:55.093829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:55.093541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:55.093829 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:55.093676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:55.093829 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:55.093775    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:55.093979 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:55.093845    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:57.093310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:57.093274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:57.093310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:57.093295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:57.093760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:57.093295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:57.093760 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:57.093421    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:23:57.093760 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:57.093538    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:57.093760 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:57.093603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:59.093060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:59.093020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:23:59.093060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:59.093043    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:23:59.093538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:23:59.093020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:23:59.093538 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:59.093147    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:23:59.093538 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:59.093199    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:23:59.093538 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:23:59.093281    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:01.092930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:01.092889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:01.092930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:01.092926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:01.092930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:01.092901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:01.093523 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:01.093034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:01.093523 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:01.093126    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:01.093523 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:01.093223    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:02.675779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:02.675710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:02.676181 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.675848    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:02.676181 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.675906    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:18.675890485 +0000 UTC m=+33.127465595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:02.776015 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:02.775986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:02.776015 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:02.776019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776124    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776135    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776147    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776157    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776179    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:24:18.776163801 +0000 UTC m=+33.227738912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:02.776209 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:02.776194    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:24:18.776184564 +0000 UTC m=+33.227759674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:03.093747 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:03.093648    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:03.093747 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:03.093692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:03.094024 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:03.093794    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:03.094024 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:03.093648    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:03.094024 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:03.093901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:03.094024 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:03.093980    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:05.093033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:05.092970    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:05.093033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:05.093006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:05.093579 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:05.093100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:05.093579 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:05.093115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:05.093579 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:05.093169    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:05.093579 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:05.093274    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:07.093754 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.093300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:07.093754 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:07.093642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:07.094376 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.094355    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:07.094576 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:07.094534    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:07.094838 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.094819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:07.095016 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:07.094981    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:07.278262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.278031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5c2x7" event={"ID":"6d8b56b1-e175-437f-bd9d-a0994baf64ad","Type":"ContainerStarted","Data":"955dc8c28739a845fd3b56800ed9a59f0b2af8089083ddec4ad3126e97d08d8d"}
Sep 08 18:24:07.281054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.281022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" event={"ID":"9b7a1699332bcb5df721e142c96ab97c","Type":"ContainerStarted","Data":"104bcfae566ee2ec5ff8e0020e93d2cc722fe26d23cf17dd911d4b0a96a79341"}
Sep 08 18:24:07.283044 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.283017    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwhsw" event={"ID":"3dce7655-4f24-49ff-a2b4-85a44d40bf5f","Type":"ContainerStarted","Data":"525c5bd006ccce2bfded683addb8a98f9efcbefc6db684ffc2f999f0267ff725"}
Sep 08 18:24:07.287445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.286081    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"acde448b32a87171a63684079aa444f826c49e5a66827b287910ca86587cd77e"}
Sep 08 18:24:07.287445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.286255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"4c2be8cc727422b9fbeba5338cdba67c514706efc84fd245bb6dad850272ae01"}
Sep 08 18:24:07.300065 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.299996    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5c2x7" podStartSLOduration=2.19612116 podStartE2EDuration="21.299982935s" totalImagesPullingTime="19.103861775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:07.299692422 +0000 UTC m=+21.751267554" watchObservedRunningTime="2026-09-08 18:24:07.299982935 +0000 UTC m=+21.751558063"
Sep 08 18:24:07.314926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.314891    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" podStartSLOduration=21.31487864 podStartE2EDuration="21.31487864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:07.314305706 +0000 UTC m=+21.765880838" watchObservedRunningTime="2026-09-08 18:24:07.31487864 +0000 UTC m=+21.766453772"
Sep 08 18:24:07.330579 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:07.330537    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hwhsw" podStartSLOduration=2.304817927 podStartE2EDuration="21.330524638s" totalImagesPullingTime="19.025706711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:07.330364601 +0000 UTC m=+21.781939733" watchObservedRunningTime="2026-09-08 18:24:07.330524638 +0000 UTC m=+21.782099771"
Sep 08 18:24:08.290709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.290673    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f2063da704829bef1181fbc5366292c" containerID="97e031db0faae5173e5dd09622d53c5155db38de9af3f1eff145b488a1f4f61c" exitCode=0
Sep 08 18:24:08.291506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.290739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"7f2063da704829bef1181fbc5366292c","Type":"ContainerDied","Data":"97e031db0faae5173e5dd09622d53c5155db38de9af3f1eff145b488a1f4f61c"}
Sep 08 18:24:08.292745 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.292698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qwcwx" event={"ID":"93a88fc2-f535-4304-ae16-cd66d5924739","Type":"ContainerStarted","Data":"fa949798acb795921d855b115647a92d9d172c48d81a8979809813e818b9f8ba"}
Sep 08 18:24:08.294519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.294487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8" event={"ID":"33c186c4-115e-4335-bb21-814c8d662096","Type":"ContainerStarted","Data":"f649dbb10f7a838896974b2e1f33f47fcfa69e2710a1139681a9e2572c6a7ff2"}
Sep 08 18:24:08.296231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.296101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pbmrz" event={"ID":"2ebeebd7-59c9-428f-bb26-025899cc0bb8","Type":"ContainerStarted","Data":"9553e6f61c8350b32601b463fd2195c5235f37a70201e08369d46585dde8495a"}
Sep 08 18:24:08.298338 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.298314    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="813d37fc8281974648bf9f55e7021a3a2d48bc42dbf8d3eaad4cae52ebbd1900" exitCode=0
Sep 08 18:24:08.298444 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.298390    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"813d37fc8281974648bf9f55e7021a3a2d48bc42dbf8d3eaad4cae52ebbd1900"}
Sep 08 18:24:08.301777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.301754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kf9x2" event={"ID":"62dbe9f7-1c35-42c3-be25-65e52c6409c4","Type":"ContainerStarted","Data":"7345b156ca2f4da8f9b196a8980f2823d7e9dbcda2b3c60f84ab1d06760b02bc"}
Sep 08 18:24:08.305157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.305133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"d454624454da339bec00ebfb54816a2d2da2d68a760b3382e2d2f4c1bc30bcb3"}
Sep 08 18:24:08.305254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.305192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"015be140965765e60c94b4397a593c0bedfcec308dbb7da7f6167e3c631b2f28"}
Sep 08 18:24:08.305254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.305235    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"da565a54928302c910b7c4859a65483269556ab0aa8bf8771d5cf48ac0ec20c8"}
Sep 08 18:24:08.305358 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.305273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"240813b8978a7a28e447dbf9e5de43fe70cdbfa653b18f349220b06ee37fe5a4"}
Sep 08 18:24:08.306578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.306559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bdp8t" event={"ID":"5fe5c088-94d4-4752-95c8-d3d4691547df","Type":"ContainerStarted","Data":"e876b88729f50d1de8a196b2b5ba97d509a3ffe8e854dd8afcca9e9e427bff26"}
Sep 08 18:24:08.357383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.357331    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qwcwx" podStartSLOduration=3.350450091 podStartE2EDuration="22.357315671s" totalImagesPullingTime="19.00686558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:08.342523698 +0000 UTC m=+22.794098830" watchObservedRunningTime="2026-09-08 18:24:08.357315671 +0000 UTC m=+22.808890804"
Sep 08 18:24:08.373185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.373131    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pbmrz" podStartSLOduration=3.362776397 podStartE2EDuration="22.373113291s" totalImagesPullingTime="19.010336894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:08.358101089 +0000 UTC m=+22.809676222" watchObservedRunningTime="2026-09-08 18:24:08.373113291 +0000 UTC m=+22.824688434"
Sep 08 18:24:08.373341 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.373298    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kf9x2" podStartSLOduration=3.3794226800000002 podStartE2EDuration="22.37329387s" totalImagesPullingTime="18.99387119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:08.372506199 +0000 UTC m=+22.824081330" watchObservedRunningTime="2026-09-08 18:24:08.37329387 +0000 UTC m=+22.824869001"
Sep 08 18:24:08.388052 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.388004    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bdp8t" podStartSLOduration=3.310040154 podStartE2EDuration="22.38798687s" totalImagesPullingTime="19.077946716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:08.387415551 +0000 UTC m=+22.838990684" watchObservedRunningTime="2026-09-08 18:24:08.38798687 +0000 UTC m=+22.839562004"
Sep 08 18:24:08.964392 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:08.964358    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 18:24:09.020488 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.020367    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T18:24:08.964388931Z","UUID":"5038692d-feff-4ba3-aba5-4e6423259676","Handler":null,"Name":"","Endpoint":""}
Sep 08 18:24:09.021850 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.021830    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 18:24:09.021958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.021862    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 18:24:09.093064 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.092975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:09.093216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.093083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:09.093216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.093094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:09.093216 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:09.093091    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:09.093216 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:09.093171    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:09.093415 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:09.093251    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:09.311762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.311708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"7f2063da704829bef1181fbc5366292c","Type":"ContainerStarted","Data":"4dfa9989232d5fec7aa02c85107424d35054a2397a5b69f14ff591235d33dd11"}
Sep 08 18:24:09.314090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:09.313893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8" event={"ID":"33c186c4-115e-4335-bb21-814c8d662096","Type":"ContainerStarted","Data":"35082e639dedf11597f0b622ec6492ee85a0ce7efd455e84c6f7d6f64465d9dc"}
Sep 08 18:24:10.321425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:10.321321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"1eff6b0a5ada2d314ba51a743b7ff90cffb24c89771373a38785b42902f2f912"}
Sep 08 18:24:10.323847 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:10.323815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8" event={"ID":"33c186c4-115e-4335-bb21-814c8d662096","Type":"ContainerStarted","Data":"bed4e6787a5db3a5d8ebef61cb16993d0199461b34a2e8a1ffdb129d8353d554"}
Sep 08 18:24:10.344616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:10.344548    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" podStartSLOduration=24.344535371 podStartE2EDuration="24.344535371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:09.327972347 +0000 UTC m=+23.779547480" watchObservedRunningTime="2026-09-08 18:24:10.344535371 +0000 UTC m=+24.796110502"
Sep 08 18:24:10.344962 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:10.344932    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xbwv8" podStartSLOduration=3.316682838 podStartE2EDuration="24.344925383s" totalImagesPullingTime="21.028242545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:10.344432932 +0000 UTC m=+24.796008063" watchObservedRunningTime="2026-09-08 18:24:10.344925383 +0000 UTC m=+24.796500544"
Sep 08 18:24:11.092821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:11.092784    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:11.093017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:11.092795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:11.093017 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:11.092907    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:11.093017 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:11.092955    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:11.093017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:11.092797    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:11.093148 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:11.093035    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:12.830470 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:12.830431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:24:12.831081 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:12.831060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:24:13.093293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.093218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:13.093293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.093249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:13.093508 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:13.093331    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:13.093508 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.093354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:13.093508 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:13.093420    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:13.093508 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:13.093492    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:13.335419 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.335372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" event={"ID":"1b01fcde-41df-450e-8ee7-f833570fa0f5","Type":"ContainerStarted","Data":"7b16c5b0a2d74f26ee7777fcfb2036a98608613496b5309cc5411b0e4daa4d0a"}
Sep 08 18:24:13.335650 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.335630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:24:13.335804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.335670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:13.336625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.336601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pbmrz"
Sep 08 18:24:13.351854 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.351798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:13.399057 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:13.398996    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx" podStartSLOduration=7.741745282 podStartE2EDuration="27.398974692s" totalImagesPullingTime="19.65722941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:13.369354141 +0000 UTC m=+27.820929273" watchObservedRunningTime="2026-09-08 18:24:13.398974692 +0000 UTC m=+27.850549825"
Sep 08 18:24:14.340091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:14.339838    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="227e258b4024b438fe215ba113af5bd5100265464ea77e6795e0571fee1068a9" exitCode=0
Sep 08 18:24:14.340530 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:14.339923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"227e258b4024b438fe215ba113af5bd5100265464ea77e6795e0571fee1068a9"}
Sep 08 18:24:14.340530 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:14.340226    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 18:24:14.341161 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:14.340655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:14.355984 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:14.355961    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:15.093383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.093352    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:15.093523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.093400    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:15.093523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.093360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:15.093523 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.093497    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:15.093679 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.093574    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:15.093679 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.093633    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:15.159288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.159168    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sb25t"]
Sep 08 18:24:15.162126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.162091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x7tgt"]
Sep 08 18:24:15.162648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.162630    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9br6q"]
Sep 08 18:24:15.344934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.344855    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="b50d41e3ed6c93aa07a9a2456b3cdba040e0f16b7bb1f0893b1de97f4e768dc0" exitCode=0
Sep 08 18:24:15.344934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.344921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"b50d41e3ed6c93aa07a9a2456b3cdba040e0f16b7bb1f0893b1de97f4e768dc0"}
Sep 08 18:24:15.345846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.344954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:15.345846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.345389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:15.345846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.345485    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 18:24:15.345846 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.345384    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:15.347235 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.347065    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:15.347235 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.347120    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:15.347346 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:15.347233    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:15.538534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:15.538506    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:16.350885 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:16.350673    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="142eab9c5a124dae473e27c2ae6245fa3daa0bbbca53e68f269a44fe905d734c" exitCode=0
Sep 08 18:24:16.351288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:16.350765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"142eab9c5a124dae473e27c2ae6245fa3daa0bbbca53e68f269a44fe905d734c"}
Sep 08 18:24:17.093453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:17.093358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:17.093610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:17.093484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:17.093610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:17.093496    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:17.093610 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:17.093494    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:17.093772 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:17.093629    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:17.093772 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:17.093745    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:18.695431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:18.695393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:18.696016 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.695529    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:18.696016 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.695605    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs podName:623bbd40-a928-4099-9d04-964fa3145559 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.69558529 +0000 UTC m=+65.147160403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs") pod "network-metrics-daemon-9br6q" (UID: "623bbd40-a928-4099-9d04-964fa3145559") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 18:24:18.795859 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:18.795810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:18.795859 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:18.795852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:18.796043 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.795966    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:18.796043 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.795997    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 18:24:18.796043 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.796017    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 18:24:18.796043 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.796027    2582 projected.go:196] Error preparing data for projected volume kube-api-access-dk75c for pod openshift-network-diagnostics/network-check-target-x7tgt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:18.796043 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.796035    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret podName:6e00ad94-0553-41e4-adf2-d9c5971cc0ab nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.796020046 +0000 UTC m=+65.247595160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret") pod "global-pull-secret-syncer-sb25t" (UID: "6e00ad94-0553-41e4-adf2-d9c5971cc0ab") : object "kube-system"/"original-pull-secret" not registered
Sep 08 18:24:18.796221 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:18.796075    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c podName:f38e3065-3591-49d0-a17a-553d326bc57d nodeName:}" failed. No retries permitted until 2026-09-08 18:24:50.79606356 +0000 UTC m=+65.247638669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dk75c" (UniqueName: "kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c") pod "network-check-target-x7tgt" (UID: "f38e3065-3591-49d0-a17a-553d326bc57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 18:24:19.093554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:19.093482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:19.093692 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:19.093483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:19.093692 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:19.093588    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:19.093692 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:19.093670    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:19.093692 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:19.093483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:19.093836 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:19.093778    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:21.093554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.093520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:21.094061 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.093561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:21.094061 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.093561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:21.094061 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:21.093637    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sb25t" podUID="6e00ad94-0553-41e4-adf2-d9c5971cc0ab"
Sep 08 18:24:21.094061 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:21.093705    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x7tgt" podUID="f38e3065-3591-49d0-a17a-553d326bc57d"
Sep 08 18:24:21.094061 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:21.093814    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9br6q" podUID="623bbd40-a928-4099-9d04-964fa3145559"
Sep 08 18:24:21.304979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.304911    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeReady"
Sep 08 18:24:21.305099 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.305053    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 18:24:21.355353 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.355320    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-66zzj"]
Sep 08 18:24:21.365340 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.365308    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="e6605793545fb7aecc6c51850d0e2cde0f08b22b4969ac32be756949148fd65a" exitCode=0
Sep 08 18:24:21.369775 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.369754    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjwrf"]
Sep 08 18:24:21.369896 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.369850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.372844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.372823    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vbqmk\""
Sep 08 18:24:21.373103 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.373088    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 18:24:21.373391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.373374    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 18:24:21.384005 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.383983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjwrf"]
Sep 08 18:24:21.384078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.384013    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66zzj"]
Sep 08 18:24:21.384078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.384033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"e6605793545fb7aecc6c51850d0e2cde0f08b22b4969ac32be756949148fd65a"}
Sep 08 18:24:21.384158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.384105    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.387767 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.387750    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 18:24:21.387907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.387895    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 18:24:21.388069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.388053    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jndxk\""
Sep 08 18:24:21.388154 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.388066    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 18:24:21.513473 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0861cf4-0c21-4246-b7b6-d36e6a654a02-cert\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.513603 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129a20f-63a5-46ff-94f2-73542d655a22-tmp-dir\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.513603 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6gct\" (UniqueName: \"kubernetes.io/projected/b129a20f-63a5-46ff-94f2-73542d655a22-kube-api-access-j6gct\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.513715 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.513715 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49bz9\" (UniqueName: \"kubernetes.io/projected/c0861cf4-0c21-4246-b7b6-d36e6a654a02-kube-api-access-49bz9\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.513857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.513743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b129a20f-63a5-46ff-94f2-73542d655a22-config-volume\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.614988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.614955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.615136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49bz9\" (UniqueName: \"kubernetes.io/projected/c0861cf4-0c21-4246-b7b6-d36e6a654a02-kube-api-access-49bz9\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.615136 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:21.615111    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:21.615241 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b129a20f-63a5-46ff-94f2-73542d655a22-config-volume\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.615291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615268    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0861cf4-0c21-4246-b7b6-d36e6a654a02-cert\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.615339 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:21.615291    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls podName:b129a20f-63a5-46ff-94f2-73542d655a22 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:22.115269229 +0000 UTC m=+36.566844343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls") pod "dns-default-66zzj" (UID: "b129a20f-63a5-46ff-94f2-73542d655a22") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:21.615397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129a20f-63a5-46ff-94f2-73542d655a22-tmp-dir\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.615397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6gct\" (UniqueName: \"kubernetes.io/projected/b129a20f-63a5-46ff-94f2-73542d655a22-kube-api-access-j6gct\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.615618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b129a20f-63a5-46ff-94f2-73542d655a22-config-volume\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.615667 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.615638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129a20f-63a5-46ff-94f2-73542d655a22-tmp-dir\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.619523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.619418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0861cf4-0c21-4246-b7b6-d36e6a654a02-cert\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.626009 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.625987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49bz9\" (UniqueName: \"kubernetes.io/projected/c0861cf4-0c21-4246-b7b6-d36e6a654a02-kube-api-access-49bz9\") pod \"ingress-canary-sjwrf\" (UID: \"c0861cf4-0c21-4246-b7b6-d36e6a654a02\") " pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:21.626103 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.626010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6gct\" (UniqueName: \"kubernetes.io/projected/b129a20f-63a5-46ff-94f2-73542d655a22-kube-api-access-j6gct\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:21.692268 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:21.692238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjwrf"
Sep 08 18:24:22.011260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:22.011231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjwrf"]
Sep 08 18:24:22.015833 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:22.015798    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0861cf4_0c21_4246_b7b6_d36e6a654a02.slice/crio-bfb26e0203df89b2daaee852d5b369937f478f65844d332ca89b2ea8028f0ef2 WatchSource:0}: Error finding container bfb26e0203df89b2daaee852d5b369937f478f65844d332ca89b2ea8028f0ef2: Status 404 returned error can't find the container with id bfb26e0203df89b2daaee852d5b369937f478f65844d332ca89b2ea8028f0ef2
Sep 08 18:24:22.119040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:22.119005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:22.119535 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:22.119149    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:22.119535 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:22.119209    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls podName:b129a20f-63a5-46ff-94f2-73542d655a22 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:23.119194669 +0000 UTC m=+37.570769783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls") pod "dns-default-66zzj" (UID: "b129a20f-63a5-46ff-94f2-73542d655a22") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:22.370923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:22.370840    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c7acc60-35eb-4357-bc47-5bfed8c77926" containerID="e0116f8f5a71d59cd9d721510ea22bc469219cfa1e33b8e33979f3c92cb90ba3" exitCode=0
Sep 08 18:24:22.370923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:22.370917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerDied","Data":"e0116f8f5a71d59cd9d721510ea22bc469219cfa1e33b8e33979f3c92cb90ba3"}
Sep 08 18:24:22.372079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:22.372052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjwrf" event={"ID":"c0861cf4-0c21-4246-b7b6-d36e6a654a02","Type":"ContainerStarted","Data":"bfb26e0203df89b2daaee852d5b369937f478f65844d332ca89b2ea8028f0ef2"}
Sep 08 18:24:23.092805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.092770    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:23.092986 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.092813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:23.092986 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.092876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:23.095949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.095922    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:24:23.097240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.097218    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 18:24:23.097361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.097271    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:24:23.097425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.097364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frqcg\""
Sep 08 18:24:23.097531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.097504    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 18:24:23.103938 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.103917    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jmlkr\""
Sep 08 18:24:23.125498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.125464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:23.126198 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:23.125581    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:23.126198 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:23.125652    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls podName:b129a20f-63a5-46ff-94f2-73542d655a22 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:25.125633234 +0000 UTC m=+39.577208366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls") pod "dns-default-66zzj" (UID: "b129a20f-63a5-46ff-94f2-73542d655a22") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:23.378666 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.378580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" event={"ID":"6c7acc60-35eb-4357-bc47-5bfed8c77926","Type":"ContainerStarted","Data":"9aba60e50b5819144ef52ecf2214ac481fc6f609a524651e9e44aef4f67aa334"}
Sep 08 18:24:23.401397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:23.401345    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8r4mg" podStartSLOduration=7.038835567 podStartE2EDuration="37.401327397s" totalImagesPullingTime="30.36249183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:23.400830355 +0000 UTC m=+37.852405487" watchObservedRunningTime="2026-09-08 18:24:23.401327397 +0000 UTC m=+37.852902529"
Sep 08 18:24:24.319373 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.319337    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bj964"]
Sep 08 18:24:24.332072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.332049    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.336626 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.336602    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srf5g\""
Sep 08 18:24:24.336941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.336920    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 18:24:24.336941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.336934    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 18:24:24.337095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.336936    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 18:24:24.337095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.337051    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 18:24:24.337833 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.337811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bj964"]
Sep 08 18:24:24.435158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.435121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98277700-ec42-487d-8021-e7ce8c97a640-crio-socket\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.435317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.435173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.435317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.435200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.435317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.435225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ks5\" (UniqueName: \"kubernetes.io/projected/98277700-ec42-487d-8021-e7ce8c97a640-kube-api-access-92ks5\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.435317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.435272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98277700-ec42-487d-8021-e7ce8c97a640-data-volume\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.536535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.536489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.536644 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:24.536583    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:24.536644 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.536623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.536747 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:24.536646    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:25.036631898 +0000 UTC m=+39.488207008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:24.536747 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.536670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ks5\" (UniqueName: \"kubernetes.io/projected/98277700-ec42-487d-8021-e7ce8c97a640-kube-api-access-92ks5\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.536747 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:24.536698    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:24.536893 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.536758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98277700-ec42-487d-8021-e7ce8c97a640-data-volume\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.536893 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:24.536768    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:25.036753411 +0000 UTC m=+39.488328546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:24.536893 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.536861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98277700-ec42-487d-8021-e7ce8c97a640-crio-socket\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.537106 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.537086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/98277700-ec42-487d-8021-e7ce8c97a640-crio-socket\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.537161 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.537106    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/98277700-ec42-487d-8021-e7ce8c97a640-data-volume\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:24.546058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:24.546030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ks5\" (UniqueName: \"kubernetes.io/projected/98277700-ec42-487d-8021-e7ce8c97a640-kube-api-access-92ks5\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:25.041778 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:25.041511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:25.041778 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.041643    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:25.041996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:25.041693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:25.041996 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.041783    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:25.041996 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.041785    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.0417626 +0000 UTC m=+40.493337710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:25.041996 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.041868    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:26.041851795 +0000 UTC m=+40.493426905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:25.142566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:25.142534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:25.142763 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.142690    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:25.142803 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:25.142778    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls podName:b129a20f-63a5-46ff-94f2-73542d655a22 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:29.142760032 +0000 UTC m=+43.594335141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls") pod "dns-default-66zzj" (UID: "b129a20f-63a5-46ff-94f2-73542d655a22") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:25.385086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:25.385003    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjwrf" event={"ID":"c0861cf4-0c21-4246-b7b6-d36e6a654a02","Type":"ContainerStarted","Data":"9826dcf228a9dda9b866dc7e5a280d2acd2fb65aac7c424c2fdfb4cfb5cecc15"}
Sep 08 18:24:25.404043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:25.403994    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjwrf" podStartSLOduration=1.722566255 podStartE2EDuration="4.403980027s" totalImagesPullingTime="2.681413772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:25.403018595 +0000 UTC m=+39.854593730" watchObservedRunningTime="2026-09-08 18:24:25.403980027 +0000 UTC m=+39.855555158"
Sep 08 18:24:26.049476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.049441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:26.049476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.049475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:26.049753 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:26.049570    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:26.049753 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:26.049577    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:26.049753 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:26.049624    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:28.049611192 +0000 UTC m=+42.501186301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:26.049753 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:26.049646    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:28.049629379 +0000 UTC m=+42.501204506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:26.454951 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.454925    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwcwx_93a88fc2-f535-4304-ae16-cd66d5924739/dns-node-resolver/0.log"
Sep 08 18:24:26.505756 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.505705    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"]
Sep 08 18:24:26.531242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.531210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"]
Sep 08 18:24:26.531391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.531324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"
Sep 08 18:24:26.534152 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.534125    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 18:24:26.534152 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.534139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 18:24:26.534326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.534125    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2dgwj\""
Sep 08 18:24:26.655430 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.655396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cnf5\" (UniqueName: \"kubernetes.io/projected/25b71bdb-9747-453b-a23d-a6c11b8ad390-kube-api-access-8cnf5\") pod \"migrator-fdbb8fdff-ghlrf\" (UID: \"25b71bdb-9747-453b-a23d-a6c11b8ad390\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"
Sep 08 18:24:26.756339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.756261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cnf5\" (UniqueName: \"kubernetes.io/projected/25b71bdb-9747-453b-a23d-a6c11b8ad390-kube-api-access-8cnf5\") pod \"migrator-fdbb8fdff-ghlrf\" (UID: \"25b71bdb-9747-453b-a23d-a6c11b8ad390\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"
Sep 08 18:24:26.765290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.765254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cnf5\" (UniqueName: \"kubernetes.io/projected/25b71bdb-9747-453b-a23d-a6c11b8ad390-kube-api-access-8cnf5\") pod \"migrator-fdbb8fdff-ghlrf\" (UID: \"25b71bdb-9747-453b-a23d-a6c11b8ad390\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"
Sep 08 18:24:26.840156 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:26.840128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"
Sep 08 18:24:27.128666 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:27.128584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf"]
Sep 08 18:24:27.131129 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:27.131098    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b71bdb_9747_453b_a23d_a6c11b8ad390.slice/crio-198bec943b5180f3fab706169b680e88b4f577e0ee5f39f2c6149787e80d3436 WatchSource:0}: Error finding container 198bec943b5180f3fab706169b680e88b4f577e0ee5f39f2c6149787e80d3436: Status 404 returned error can't find the container with id 198bec943b5180f3fab706169b680e88b4f577e0ee5f39f2c6149787e80d3436
Sep 08 18:24:27.390854 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:27.390822    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf" event={"ID":"25b71bdb-9747-453b-a23d-a6c11b8ad390","Type":"ContainerStarted","Data":"198bec943b5180f3fab706169b680e88b4f577e0ee5f39f2c6149787e80d3436"}
Sep 08 18:24:27.449148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:27.449122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kf9x2_62dbe9f7-1c35-42c3-be25-65e52c6409c4/node-ca/0.log"
Sep 08 18:24:28.065921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.065882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:28.065921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.065922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:28.066444 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:28.066056    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:28.066444 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:28.066141    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:32.06611952 +0000 UTC m=+46.517694642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:28.066444 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:28.066063    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:28.066444 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:28.066196    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:32.066183866 +0000 UTC m=+46.517758977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:28.253785 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.253760    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjwrf_c0861cf4-0c21-4246-b7b6-d36e6a654a02/serve-healthcheck-canary/0.log"
Sep 08 18:24:28.283071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.283031    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-v6gdw"]
Sep 08 18:24:28.302961 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.302934    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-v6gdw"]
Sep 08 18:24:28.303100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.303024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.306136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.306110    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-m645n\""
Sep 08 18:24:28.306286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.306113    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 18:24:28.307264 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.307240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 18:24:28.307500 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.307332    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 18:24:28.307500 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.307234    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 18:24:28.470401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.470359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-key\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.470401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.470397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmxm\" (UniqueName: \"kubernetes.io/projected/449a5416-ede7-4ed7-805d-e1cdaa37bbee-kube-api-access-jkmxm\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.470614 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.470431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-cabundle\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.571671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.571634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-key\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.571671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.571670    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmxm\" (UniqueName: \"kubernetes.io/projected/449a5416-ede7-4ed7-805d-e1cdaa37bbee-kube-api-access-jkmxm\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.571879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.571698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-cabundle\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.572387 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.572360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-cabundle\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.574279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.574263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/449a5416-ede7-4ed7-805d-e1cdaa37bbee-signing-key\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.579934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.579915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmxm\" (UniqueName: \"kubernetes.io/projected/449a5416-ede7-4ed7-805d-e1cdaa37bbee-kube-api-access-jkmxm\") pod \"service-ca-75d99c484-v6gdw\" (UID: \"449a5416-ede7-4ed7-805d-e1cdaa37bbee\") " pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.612949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.612917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-v6gdw"
Sep 08 18:24:28.893565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:28.893535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-v6gdw"]
Sep 08 18:24:28.896380 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:28.896349    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449a5416_ede7_4ed7_805d_e1cdaa37bbee.slice/crio-01df4ad3247194b40347d91ab6d0a8b95672bae45ff487c000ea34b0f5b9b3e4 WatchSource:0}: Error finding container 01df4ad3247194b40347d91ab6d0a8b95672bae45ff487c000ea34b0f5b9b3e4: Status 404 returned error can't find the container with id 01df4ad3247194b40347d91ab6d0a8b95672bae45ff487c000ea34b0f5b9b3e4
Sep 08 18:24:29.177388 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:29.177355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:29.177762 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:29.177508    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 18:24:29.177762 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:29.177576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls podName:b129a20f-63a5-46ff-94f2-73542d655a22 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:37.177561092 +0000 UTC m=+51.629136207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls") pod "dns-default-66zzj" (UID: "b129a20f-63a5-46ff-94f2-73542d655a22") : secret "dns-default-metrics-tls" not found
Sep 08 18:24:29.398527 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:29.398490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf" event={"ID":"25b71bdb-9747-453b-a23d-a6c11b8ad390","Type":"ContainerStarted","Data":"f7371609ffca2386c838ef1b72128447e6faa2ec002677cf28f3148191a19b71"}
Sep 08 18:24:29.398693 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:29.398563    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf" event={"ID":"25b71bdb-9747-453b-a23d-a6c11b8ad390","Type":"ContainerStarted","Data":"5b320ea536eecfa4b9bbb0dfccf637d41a597fb8e35f3bf50e7925ccf74add2b"}
Sep 08 18:24:29.400126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:29.400097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-v6gdw" event={"ID":"449a5416-ede7-4ed7-805d-e1cdaa37bbee","Type":"ContainerStarted","Data":"01df4ad3247194b40347d91ab6d0a8b95672bae45ff487c000ea34b0f5b9b3e4"}
Sep 08 18:24:29.417692 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:29.417633    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ghlrf" podStartSLOduration=1.6668186710000001 podStartE2EDuration="3.417616128s" totalImagesPullingTime="1.750797457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:29.416869977 +0000 UTC m=+43.868445120" watchObservedRunningTime="2026-09-08 18:24:29.417616128 +0000 UTC m=+43.869191259"
Sep 08 18:24:31.407697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:31.407658    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-v6gdw" event={"ID":"449a5416-ede7-4ed7-805d-e1cdaa37bbee","Type":"ContainerStarted","Data":"1b3a40818b7851ba3c7fad5546d7c605a4b427d615ec603450960dbd5491f9b8"}
Sep 08 18:24:31.426867 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:31.426815    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-v6gdw" podStartSLOduration=1.043091064 podStartE2EDuration="3.42679911s" totalImagesPullingTime="2.383708046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:31.426112377 +0000 UTC m=+45.877687508" watchObservedRunningTime="2026-09-08 18:24:31.42679911 +0000 UTC m=+45.878374242"
Sep 08 18:24:32.098157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:32.098123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:32.098353 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:32.098164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:32.098353 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:32.098269    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 18:24:32.098353 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:32.098344    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:40.098324669 +0000 UTC m=+54.549899792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : configmap "kube-rbac-proxy" not found
Sep 08 18:24:32.098509 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:32.098355    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:32.098509 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:32.098408    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:40.098392851 +0000 UTC m=+54.549967961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:37.238656 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:37.238618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:37.241166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:37.241141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b129a20f-63a5-46ff-94f2-73542d655a22-metrics-tls\") pod \"dns-default-66zzj\" (UID: \"b129a20f-63a5-46ff-94f2-73542d655a22\") " pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:37.281305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:37.281272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:37.567182 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:37.567075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-66zzj"]
Sep 08 18:24:37.570209 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:37.570181    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb129a20f_63a5_46ff_94f2_73542d655a22.slice/crio-d79829f31e2213ebbf393e39601e468ef4cbaa16054bf5612d66946699a8fe7d WatchSource:0}: Error finding container d79829f31e2213ebbf393e39601e468ef4cbaa16054bf5612d66946699a8fe7d: Status 404 returned error can't find the container with id d79829f31e2213ebbf393e39601e468ef4cbaa16054bf5612d66946699a8fe7d
Sep 08 18:24:38.427608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:38.427559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66zzj" event={"ID":"b129a20f-63a5-46ff-94f2-73542d655a22","Type":"ContainerStarted","Data":"d79829f31e2213ebbf393e39601e468ef4cbaa16054bf5612d66946699a8fe7d"}
Sep 08 18:24:40.163321 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.163284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:40.163321 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.163321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:40.163767 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:40.163431    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:40.163767 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:24:40.163501    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls podName:98277700-ec42-487d-8021-e7ce8c97a640 nodeName:}" failed. No retries permitted until 2026-09-08 18:24:56.163482738 +0000 UTC m=+70.615057870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bj964" (UID: "98277700-ec42-487d-8021-e7ce8c97a640") : secret "insights-runtime-extractor-tls" not found
Sep 08 18:24:40.163955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.163935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/98277700-ec42-487d-8021-e7ce8c97a640-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:40.434555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.434466    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66zzj" event={"ID":"b129a20f-63a5-46ff-94f2-73542d655a22","Type":"ContainerStarted","Data":"74c68bddf83db51b2bf61bd3340f183728dd8860443a27571b939596f6dd0dcc"}
Sep 08 18:24:40.434555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.434519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-66zzj" event={"ID":"b129a20f-63a5-46ff-94f2-73542d655a22","Type":"ContainerStarted","Data":"5fc26de5ca99fac07bb5bfd129a02760016894426dd2803e27e327383acbffc3"}
Sep 08 18:24:40.434807 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.434600    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:40.455031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:40.454968    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-66zzj" podStartSLOduration=17.68176469 podStartE2EDuration="19.454953729s" totalImagesPullingTime="1.773189039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:40.453516063 +0000 UTC m=+54.905091194" watchObservedRunningTime="2026-09-08 18:24:40.454953729 +0000 UTC m=+54.906528855"
Sep 08 18:24:47.365612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.365577    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xlpjx"
Sep 08 18:24:47.826337 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.826300    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"]
Sep 08 18:24:47.829107 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.829081    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:47.829651 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.829629    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"]
Sep 08 18:24:47.831246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.831232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.831710 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.831693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9kgm5\""
Sep 08 18:24:47.831919 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.831907    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 18:24:47.833293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.833273    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 18:24:47.833384 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.833274    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 18:24:47.833384 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.833272    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 18:24:47.834278 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.834256    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 18:24:47.834385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.834364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 18:24:47.834781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.834762    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 18:24:47.834885 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.834771    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 18:24:47.840423 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.840398    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"]
Sep 08 18:24:47.849035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.849015    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"]
Sep 08 18:24:47.919529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78z6f\" (UniqueName: \"kubernetes.io/projected/153b3c6f-1af9-487b-909a-e3983050212b-kube-api-access-78z6f\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:47.919529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.919743 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d351281e-43fd-472a-9305-a02b30f2f656-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.919743 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/153b3c6f-1af9-487b-909a-e3983050212b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:47.919743 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.919743 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfhx\" (UniqueName: \"kubernetes.io/projected/d351281e-43fd-472a-9305-a02b30f2f656-kube-api-access-hzfhx\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.919894 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919768    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:47.919894 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:47.919791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020515 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78z6f\" (UniqueName: \"kubernetes.io/projected/153b3c6f-1af9-487b-909a-e3983050212b-kube-api-access-78z6f\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:48.020515 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020740 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d351281e-43fd-472a-9305-a02b30f2f656-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/153b3c6f-1af9-487b-909a-e3983050212b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:48.020789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020893 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfhx\" (UniqueName: \"kubernetes.io/projected/d351281e-43fd-472a-9305-a02b30f2f656-kube-api-access-hzfhx\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020947 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.020998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.020985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.021601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.021574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d351281e-43fd-472a-9305-a02b30f2f656-additional-service-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.023213 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.023193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.023854 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.023831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-ca\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.023945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.023902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-hub\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.024019 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.024003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/153b3c6f-1af9-487b-909a-e3983050212b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:48.024054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.024036    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d351281e-43fd-472a-9305-a02b30f2f656-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.031633 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.031605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78z6f\" (UniqueName: \"kubernetes.io/projected/153b3c6f-1af9-487b-909a-e3983050212b-kube-api-access-78z6f\") pod \"managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb\" (UID: \"153b3c6f-1af9-487b-909a-e3983050212b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:48.033975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.033951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfhx\" (UniqueName: \"kubernetes.io/projected/d351281e-43fd-472a-9305-a02b30f2f656-kube-api-access-hzfhx\") pod \"cluster-proxy-proxy-agent-586944cc69-z6lt5\" (UID: \"d351281e-43fd-472a-9305-a02b30f2f656\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.155567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.155540    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"
Sep 08 18:24:48.159976 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.159956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"
Sep 08 18:24:48.427061 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.426981    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb"]
Sep 08 18:24:48.431141 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:48.431115    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod153b3c6f_1af9_487b_909a_e3983050212b.slice/crio-9e1a4be95c8267dcf88cdcbe073487a5e8a86dda7bb72671db8111d29d77e25b WatchSource:0}: Error finding container 9e1a4be95c8267dcf88cdcbe073487a5e8a86dda7bb72671db8111d29d77e25b: Status 404 returned error can't find the container with id 9e1a4be95c8267dcf88cdcbe073487a5e8a86dda7bb72671db8111d29d77e25b
Sep 08 18:24:48.447535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.447505    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5"]
Sep 08 18:24:48.450427 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:48.450403    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd351281e_43fd_472a_9305_a02b30f2f656.slice/crio-607f0ff1c3f4435dee7d636f3350cfa18023930e4fc1264b9beb66120736d0ff WatchSource:0}: Error finding container 607f0ff1c3f4435dee7d636f3350cfa18023930e4fc1264b9beb66120736d0ff: Status 404 returned error can't find the container with id 607f0ff1c3f4435dee7d636f3350cfa18023930e4fc1264b9beb66120736d0ff
Sep 08 18:24:48.465253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.465218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5" event={"ID":"d351281e-43fd-472a-9305-a02b30f2f656","Type":"ContainerStarted","Data":"607f0ff1c3f4435dee7d636f3350cfa18023930e4fc1264b9beb66120736d0ff"}
Sep 08 18:24:48.466536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:48.466511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb" event={"ID":"153b3c6f-1af9-487b-909a-e3983050212b","Type":"ContainerStarted","Data":"9e1a4be95c8267dcf88cdcbe073487a5e8a86dda7bb72671db8111d29d77e25b"}
Sep 08 18:24:50.441234 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.440538    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-66zzj"
Sep 08 18:24:50.745531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.745452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:50.748547 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.748508    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 18:24:50.758975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.758945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/623bbd40-a928-4099-9d04-964fa3145559-metrics-certs\") pod \"network-metrics-daemon-9br6q\" (UID: \"623bbd40-a928-4099-9d04-964fa3145559\") " pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:50.846316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.846278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:50.846316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.846321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:50.849580 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.849552    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 18:24:50.849779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.849629    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 18:24:50.859578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.859431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 18:24:50.859679 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.859616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e00ad94-0553-41e4-adf2-d9c5971cc0ab-original-pull-secret\") pod \"global-pull-secret-syncer-sb25t\" (UID: \"6e00ad94-0553-41e4-adf2-d9c5971cc0ab\") " pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:50.870502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:50.870476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk75c\" (UniqueName: \"kubernetes.io/projected/f38e3065-3591-49d0-a17a-553d326bc57d-kube-api-access-dk75c\") pod \"network-check-target-x7tgt\" (UID: \"f38e3065-3591-49d0-a17a-553d326bc57d\") " pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:51.007910 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:51.007828    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jmlkr\""
Sep 08 18:24:51.015085 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:51.015059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:51.015222 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:51.015179    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-frqcg\""
Sep 08 18:24:51.018545 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:51.018520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sb25t"
Sep 08 18:24:51.023383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:51.023360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9br6q"
Sep 08 18:24:52.782519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:52.782439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9br6q"]
Sep 08 18:24:52.785997 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:52.785965    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623bbd40_a928_4099_9d04_964fa3145559.slice/crio-b3f40329040f63b4c3059e11ac83aff8934d8a29074e2ca253398aae96b93eb4 WatchSource:0}: Error finding container b3f40329040f63b4c3059e11ac83aff8934d8a29074e2ca253398aae96b93eb4: Status 404 returned error can't find the container with id b3f40329040f63b4c3059e11ac83aff8934d8a29074e2ca253398aae96b93eb4
Sep 08 18:24:52.812054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:52.812032    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x7tgt"]
Sep 08 18:24:52.814650 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:52.814625    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38e3065_3591_49d0_a17a_553d326bc57d.slice/crio-756905f5cdff3d6161d9d58621d1d2bba49cdbd26d136ccdd3cbc6aa43261250 WatchSource:0}: Error finding container 756905f5cdff3d6161d9d58621d1d2bba49cdbd26d136ccdd3cbc6aa43261250: Status 404 returned error can't find the container with id 756905f5cdff3d6161d9d58621d1d2bba49cdbd26d136ccdd3cbc6aa43261250
Sep 08 18:24:52.815030 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:52.815010    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sb25t"]
Sep 08 18:24:52.817856 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:52.817836    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e00ad94_0553_41e4_adf2_d9c5971cc0ab.slice/crio-5565ca57b66c55e9e163f7b3694502f18959b2b475df07ae8d1193a1a5aa3bc0 WatchSource:0}: Error finding container 5565ca57b66c55e9e163f7b3694502f18959b2b475df07ae8d1193a1a5aa3bc0: Status 404 returned error can't find the container with id 5565ca57b66c55e9e163f7b3694502f18959b2b475df07ae8d1193a1a5aa3bc0
Sep 08 18:24:53.490943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.490817    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5" event={"ID":"d351281e-43fd-472a-9305-a02b30f2f656","Type":"ContainerStarted","Data":"134bffe1fb476544a86a1227914b8c5d177a41c2e17b882438a71508b755710e"}
Sep 08 18:24:53.490943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.490890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5" event={"ID":"d351281e-43fd-472a-9305-a02b30f2f656","Type":"ContainerStarted","Data":"5fe32f3034d24c1397652aa6c707f20d8718f84bca832377202702c83250fa93"}
Sep 08 18:24:53.490943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.490908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5" event={"ID":"d351281e-43fd-472a-9305-a02b30f2f656","Type":"ContainerStarted","Data":"422ddc81d37c30e26e76003a0aa3b1f5edb581e7b5641ed51c2628401d947ffd"}
Sep 08 18:24:53.493895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.493859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb" event={"ID":"153b3c6f-1af9-487b-909a-e3983050212b","Type":"ContainerStarted","Data":"dae9a7be286be8971de34441d374d0e9a37c5385a7676fdcc82bcbb80555729f"}
Sep 08 18:24:53.498765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.498666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9br6q" event={"ID":"623bbd40-a928-4099-9d04-964fa3145559","Type":"ContainerStarted","Data":"b3f40329040f63b4c3059e11ac83aff8934d8a29074e2ca253398aae96b93eb4"}
Sep 08 18:24:53.500434 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.500377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sb25t" event={"ID":"6e00ad94-0553-41e4-adf2-d9c5971cc0ab","Type":"ContainerStarted","Data":"5565ca57b66c55e9e163f7b3694502f18959b2b475df07ae8d1193a1a5aa3bc0"}
Sep 08 18:24:53.502790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.502762    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x7tgt" event={"ID":"f38e3065-3591-49d0-a17a-553d326bc57d","Type":"ContainerStarted","Data":"756905f5cdff3d6161d9d58621d1d2bba49cdbd26d136ccdd3cbc6aa43261250"}
Sep 08 18:24:53.511509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.511454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-586944cc69-z6lt5" podStartSLOduration=2.432128208 podStartE2EDuration="6.51143583s" totalImagesPullingTime="4.079307622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:53.509480873 +0000 UTC m=+67.961056004" watchObservedRunningTime="2026-09-08 18:24:53.51143583 +0000 UTC m=+67.963010963"
Sep 08 18:24:53.525628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:53.525578    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75f9f9f4c4-9pvfb" podStartSLOduration=2.444387743 podStartE2EDuration="6.525560918s" totalImagesPullingTime="4.081173175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:53.52437347 +0000 UTC m=+67.975948601" watchObservedRunningTime="2026-09-08 18:24:53.525560918 +0000 UTC m=+67.977136050"
Sep 08 18:24:54.510146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:54.510105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9br6q" event={"ID":"623bbd40-a928-4099-9d04-964fa3145559","Type":"ContainerStarted","Data":"b3017bdc51a7af7a8a02fd7795140a3759d77688e97f3b1e087e82e0e9a4d6c2"}
Sep 08 18:24:54.510589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:54.510176    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9br6q" event={"ID":"623bbd40-a928-4099-9d04-964fa3145559","Type":"ContainerStarted","Data":"82c588db14b901a38576cede4599401044f6f8ede893b803ebfeeddfe33ec713"}
Sep 08 18:24:54.528073 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:54.528016    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9br6q" podStartSLOduration=67.443723061 podStartE2EDuration="1m8.528002062s" totalImagesPullingTime="1.084279001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:54.526667705 +0000 UTC m=+68.978242839" watchObservedRunningTime="2026-09-08 18:24:54.528002062 +0000 UTC m=+68.979577220"
Sep 08 18:24:56.194069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:56.194030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:56.196951 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:56.196927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/98277700-ec42-487d-8021-e7ce8c97a640-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bj964\" (UID: \"98277700-ec42-487d-8021-e7ce8c97a640\") " pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:56.444990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:56.444914    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-srf5g\""
Sep 08 18:24:56.452356 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:56.452331    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bj964"
Sep 08 18:24:57.698989 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:57.698896    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bj964"]
Sep 08 18:24:57.701698 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:24:57.701669    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98277700_ec42_487d_8021_e7ce8c97a640.slice/crio-492d7a656de5c91f6b78d8f5aa3e51a4387cf3be2e646767dfe6dd89aa5abe66 WatchSource:0}: Error finding container 492d7a656de5c91f6b78d8f5aa3e51a4387cf3be2e646767dfe6dd89aa5abe66: Status 404 returned error can't find the container with id 492d7a656de5c91f6b78d8f5aa3e51a4387cf3be2e646767dfe6dd89aa5abe66
Sep 08 18:24:58.534602 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.533796    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x7tgt" event={"ID":"f38e3065-3591-49d0-a17a-553d326bc57d","Type":"ContainerStarted","Data":"effcb33df71003e8e4c3d12b6b94a92c371c7f895abf8db77f7e972634c4e3fa"}
Sep 08 18:24:58.534602 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.534543    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:24:58.536941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.536908    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sb25t" event={"ID":"6e00ad94-0553-41e4-adf2-d9c5971cc0ab","Type":"ContainerStarted","Data":"dd1dc05d5c6a3a12edb59db92c7ed73d40e275fa3c8c7a28f723625b38202787"}
Sep 08 18:24:58.538667 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.538646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bj964" event={"ID":"98277700-ec42-487d-8021-e7ce8c97a640","Type":"ContainerStarted","Data":"be7ee00761b3e055f7bb8a7c74e3ca7d8f952985ef25f26bb37773391e1b8855"}
Sep 08 18:24:58.538786 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.538687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bj964" event={"ID":"98277700-ec42-487d-8021-e7ce8c97a640","Type":"ContainerStarted","Data":"d04dbd7ea3522282ac2ee2663da494046de650958dcf22289078a5bcb6e286f2"}
Sep 08 18:24:58.538786 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.538713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bj964" event={"ID":"98277700-ec42-487d-8021-e7ce8c97a640","Type":"ContainerStarted","Data":"492d7a656de5c91f6b78d8f5aa3e51a4387cf3be2e646767dfe6dd89aa5abe66"}
Sep 08 18:24:58.551452 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.551402    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x7tgt" podStartSLOduration=67.935920047 podStartE2EDuration="1m12.55138794s" totalImagesPullingTime="4.615467893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:58.550162287 +0000 UTC m=+73.001737419" watchObservedRunningTime="2026-09-08 18:24:58.55138794 +0000 UTC m=+73.002963073"
Sep 08 18:24:58.565061 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:24:58.565013    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sb25t" podStartSLOduration=67.940972611 podStartE2EDuration="1m12.564996636s" totalImagesPullingTime="4.624024025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:24:58.564002248 +0000 UTC m=+73.015577383" watchObservedRunningTime="2026-09-08 18:24:58.564996636 +0000 UTC m=+73.016571767"
Sep 08 18:25:00.546949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:00.546912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bj964" event={"ID":"98277700-ec42-487d-8021-e7ce8c97a640","Type":"ContainerStarted","Data":"c41a9c7de5057de164bc12e87b437a6a06bf8086fff76a9cb19f5ffa3564469c"}
Sep 08 18:25:00.571526 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:00.571477    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bj964" podStartSLOduration=34.521633697 podStartE2EDuration="36.571460715s" totalImagesPullingTime="2.049827018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:00.570852654 +0000 UTC m=+75.022427789" watchObservedRunningTime="2026-09-08 18:25:00.571460715 +0000 UTC m=+75.023035846"
Sep 08 18:25:11.500322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.500194    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-72rsq"]
Sep 08 18:25:11.505529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.505506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.508356 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.508335    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 18:25:11.508857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.508831    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 18:25:11.509110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.509094    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 18:25:11.510034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.510013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 18:25:11.510223 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.510203    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f4lkk\""
Sep 08 18:25:11.510368 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.510305    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 18:25:11.510498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.510029    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 18:25:11.594522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5hc\" (UniqueName: \"kubernetes.io/projected/e9a26264-f9d3-4ce6-945b-81ab8e926f17-kube-api-access-jd5hc\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-textfile\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594539    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-tls\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594592    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-root\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-sys\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-wtmp\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.594766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.594738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-metrics-client-ca\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695348 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-tls\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695348 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-root\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-sys\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-wtmp\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695451    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-root\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-sys\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-metrics-client-ca\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5hc\" (UniqueName: \"kubernetes.io/projected/e9a26264-f9d3-4ce6-945b-81ab8e926f17-kube-api-access-jd5hc\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.695586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-textfile\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.696001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-wtmp\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.696001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.695982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-textfile\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.696324 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.696304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-metrics-client-ca\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.696377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.696357    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-accelerators-collector-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.697995 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.697967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.698103 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.698043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e9a26264-f9d3-4ce6-945b-81ab8e926f17-node-exporter-tls\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.705525 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.705488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5hc\" (UniqueName: \"kubernetes.io/projected/e9a26264-f9d3-4ce6-945b-81ab8e926f17-kube-api-access-jd5hc\") pod \"node-exporter-72rsq\" (UID: \"e9a26264-f9d3-4ce6-945b-81ab8e926f17\") " pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.815674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:11.815603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-72rsq"
Sep 08 18:25:11.826105 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:25:11.826080    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a26264_f9d3_4ce6_945b_81ab8e926f17.slice/crio-d63126b994eba4b5226ced4f454a44330192c2f5497a07d237944392ebfd783a WatchSource:0}: Error finding container d63126b994eba4b5226ced4f454a44330192c2f5497a07d237944392ebfd783a: Status 404 returned error can't find the container with id d63126b994eba4b5226ced4f454a44330192c2f5497a07d237944392ebfd783a
Sep 08 18:25:12.588952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:12.588912    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72rsq" event={"ID":"e9a26264-f9d3-4ce6-945b-81ab8e926f17","Type":"ContainerStarted","Data":"d63126b994eba4b5226ced4f454a44330192c2f5497a07d237944392ebfd783a"}
Sep 08 18:25:13.594262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:13.594230    2582 generic.go:356] "Generic (PLEG): container finished" podID="e9a26264-f9d3-4ce6-945b-81ab8e926f17" containerID="9e650a63682e9734c0a4310161a60a9ce13b324b08a79260994ffcf530945f09" exitCode=0
Sep 08 18:25:13.594645 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:13.594296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72rsq" event={"ID":"e9a26264-f9d3-4ce6-945b-81ab8e926f17","Type":"ContainerDied","Data":"9e650a63682e9734c0a4310161a60a9ce13b324b08a79260994ffcf530945f09"}
Sep 08 18:25:14.603562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:14.603524    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72rsq" event={"ID":"e9a26264-f9d3-4ce6-945b-81ab8e926f17","Type":"ContainerStarted","Data":"2aca99f1cc54980cc5a3542de72f2b5c46ade3f06db2b14785e6559cd5f6011a"}
Sep 08 18:25:14.604040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:14.603625    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-72rsq" event={"ID":"e9a26264-f9d3-4ce6-945b-81ab8e926f17","Type":"ContainerStarted","Data":"bbc0f71ea946f85498b2da246ea68167f745f9c9b61adf5108a188135dc0f7c2"}
Sep 08 18:25:14.631152 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:14.631098    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-72rsq" podStartSLOduration=2.4702779 podStartE2EDuration="3.631079589s" totalImagesPullingTime="1.092226613s" totalInitContainerRuntime="68.575076ms" isStatefulPod=true podCreationTimestamp="2026-09-08 18:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:25:14.629281347 +0000 UTC m=+89.080856476" watchObservedRunningTime="2026-09-08 18:25:14.631079589 +0000 UTC m=+89.082654723"
Sep 08 18:25:30.550181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:25:30.550146    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x7tgt"
Sep 08 18:28:45.953448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:28:45.953421    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 18:29:39.090740 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.090651    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kntxk"]
Sep 08 18:29:39.092741 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.092703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.095670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.095640    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 18:29:39.095966 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.095949    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 18:29:39.096968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.096943    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xs72q\""
Sep 08 18:29:39.104186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.104160    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kntxk"]
Sep 08 18:29:39.184869 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.184832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-bound-sa-token\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.185039 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.184878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzn7m\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-kube-api-access-lzn7m\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.285886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.285834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-bound-sa-token\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.285886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.285890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzn7m\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-kube-api-access-lzn7m\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.296320 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.296288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-bound-sa-token\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.296440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.296411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzn7m\" (UniqueName: \"kubernetes.io/projected/d1968dd6-b440-4e1f-8bda-d578a7ee9baf-kube-api-access-lzn7m\") pod \"cert-manager-55b4d954d5-kntxk\" (UID: \"d1968dd6-b440-4e1f-8bda-d578a7ee9baf\") " pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.405842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.405812    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kntxk"
Sep 08 18:29:39.670438 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.670415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kntxk"]
Sep 08 18:29:39.672643 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:29:39.672613    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1968dd6_b440_4e1f_8bda_d578a7ee9baf.slice/crio-732a0bd99237c988d37278446a68050571086df028c7777591355f59ad63f488 WatchSource:0}: Error finding container 732a0bd99237c988d37278446a68050571086df028c7777591355f59ad63f488: Status 404 returned error can't find the container with id 732a0bd99237c988d37278446a68050571086df028c7777591355f59ad63f488
Sep 08 18:29:39.674517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:39.674499    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:29:40.460270 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:40.460230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kntxk" event={"ID":"d1968dd6-b440-4e1f-8bda-d578a7ee9baf","Type":"ContainerStarted","Data":"732a0bd99237c988d37278446a68050571086df028c7777591355f59ad63f488"}
Sep 08 18:29:42.228832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.228797    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"]
Sep 08 18:29:42.231073 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.231055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.235412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.235391    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 18:29:42.235538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.235395    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 18:29:42.235538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.235395    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6975g\""
Sep 08 18:29:42.240205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.240060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"]
Sep 08 18:29:42.310990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.310953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.311166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.311026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd7n7\" (UniqueName: \"kubernetes.io/projected/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-kube-api-access-vd7n7\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.411429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.411385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd7n7\" (UniqueName: \"kubernetes.io/projected/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-kube-api-access-vd7n7\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.411598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.411450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.411890 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.411867    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.420076 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.420048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd7n7\" (UniqueName: \"kubernetes.io/projected/20e7e43a-a0f9-4a7f-8cde-a5cab68927d9-kube-api-access-vd7n7\") pod \"openshift-lws-operator-fd8ccff4c-v4ddv\" (UID: \"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.543098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.543022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"
Sep 08 18:29:42.858306 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:42.858282    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv"]
Sep 08 18:29:42.860293 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:29:42.860262    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20e7e43a_a0f9_4a7f_8cde_a5cab68927d9.slice/crio-07150b245659aa2c42eddef80154c28a7bc83166e211a887961e30812f33e49d WatchSource:0}: Error finding container 07150b245659aa2c42eddef80154c28a7bc83166e211a887961e30812f33e49d: Status 404 returned error can't find the container with id 07150b245659aa2c42eddef80154c28a7bc83166e211a887961e30812f33e49d
Sep 08 18:29:43.473237 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:43.473193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv" event={"ID":"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9","Type":"ContainerStarted","Data":"07150b245659aa2c42eddef80154c28a7bc83166e211a887961e30812f33e49d"}
Sep 08 18:29:43.474759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:43.474710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kntxk" event={"ID":"d1968dd6-b440-4e1f-8bda-d578a7ee9baf","Type":"ContainerStarted","Data":"d523bb0eab29a135077e7cd630910a467db330a10810e4206ea25d547828ed2e"}
Sep 08 18:29:43.492490 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:43.492439    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kntxk" podStartSLOduration=1.537802857 podStartE2EDuration="4.49242382s" totalImagesPullingTime="2.954620963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:29:43.490536788 +0000 UTC m=+357.942111920" watchObservedRunningTime="2026-09-08 18:29:43.49242382 +0000 UTC m=+357.943998952"
Sep 08 18:29:45.486760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:45.486704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv" event={"ID":"20e7e43a-a0f9-4a7f-8cde-a5cab68927d9","Type":"ContainerStarted","Data":"c89d0aff4d4ba32226ebd7a2d83717a066dd6e60ba63d80ff59f3e8fa321caf8"}
Sep 08 18:29:45.505740 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:29:45.505674    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v4ddv" podStartSLOduration=0.992593966 podStartE2EDuration="3.505658688s" totalImagesPullingTime="2.513064722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:29:45.50343222 +0000 UTC m=+359.955007352" watchObservedRunningTime="2026-09-08 18:29:45.505658688 +0000 UTC m=+359.957233820"
Sep 08 18:30:00.804764 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.804715    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"]
Sep 08 18:30:00.806956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.806938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:00.810833 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.810808    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 18:30:00.811428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.811405    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 18:30:00.811523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.811483    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 18:30:00.811523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.811506    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w9cwx\""
Sep 08 18:30:00.812448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.812426    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 18:30:00.831932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.831899    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"]
Sep 08 18:30:00.946095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.946056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:00.946095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.946095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:00.946300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.946120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4scfm\" (UniqueName: \"kubernetes.io/projected/9ec45c56-cd0e-4e44-82af-361d05980a6b-kube-api-access-4scfm\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:00.946300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.946138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:00.946300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:00.946228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.046990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.047038    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.047064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4scfm\" (UniqueName: \"kubernetes.io/projected/9ec45c56-cd0e-4e44-82af-361d05980a6b-kube-api-access-4scfm\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.047089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.047135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.047608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.047584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-manifests\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.049460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.049425    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec45c56-cd0e-4e44-82af-361d05980a6b-tmp\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.049592 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.049572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.049682 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.049665    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ec45c56-cd0e-4e44-82af-361d05980a6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.056006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.055955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4scfm\" (UniqueName: \"kubernetes.io/projected/9ec45c56-cd0e-4e44-82af-361d05980a6b-kube-api-access-4scfm\") pod \"opendatahub-operator-controller-manager-8d688d88f-6ccqg\" (UID: \"9ec45c56-cd0e-4e44-82af-361d05980a6b\") " pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.117541 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.117502    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:01.400385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.400361    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"]
Sep 08 18:30:01.402309 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:01.402282    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec45c56_cd0e_4e44_82af_361d05980a6b.slice/crio-31efe0528790bdb6fa2af5992305d477c2f4ac4de4e43cbe4afec64847ed270b WatchSource:0}: Error finding container 31efe0528790bdb6fa2af5992305d477c2f4ac4de4e43cbe4afec64847ed270b: Status 404 returned error can't find the container with id 31efe0528790bdb6fa2af5992305d477c2f4ac4de4e43cbe4afec64847ed270b
Sep 08 18:30:01.546164 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:01.546123    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg" event={"ID":"9ec45c56-cd0e-4e44-82af-361d05980a6b","Type":"ContainerStarted","Data":"31efe0528790bdb6fa2af5992305d477c2f4ac4de4e43cbe4afec64847ed270b"}
Sep 08 18:30:05.568552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:05.568506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg" event={"ID":"9ec45c56-cd0e-4e44-82af-361d05980a6b","Type":"ContainerStarted","Data":"9b82815958ced471a69e272af5c13418ab1013e57b9cbeffcb6d539171750ad0"}
Sep 08 18:30:06.575353 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:06.575260    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ec45c56-cd0e-4e44-82af-361d05980a6b" containerID="9b82815958ced471a69e272af5c13418ab1013e57b9cbeffcb6d539171750ad0" exitCode=0
Sep 08 18:30:06.575806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:06.575760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg" event={"ID":"9ec45c56-cd0e-4e44-82af-361d05980a6b","Type":"ContainerDied","Data":"9b82815958ced471a69e272af5c13418ab1013e57b9cbeffcb6d539171750ad0"}
Sep 08 18:30:07.581248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:07.581161    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg" event={"ID":"9ec45c56-cd0e-4e44-82af-361d05980a6b","Type":"ContainerStarted","Data":"d041cfc1520e5ee2a0edf5934d8ff2571b3693161b18a005ffd98fe308d84ec9"}
Sep 08 18:30:07.581590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:07.581314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:07.608187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:07.608141    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg" podStartSLOduration=2.262799586 podStartE2EDuration="7.608126079s" totalImagesPullingTime="4.30899836s" totalInitContainerRuntime="1.036328133s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:07.605955564 +0000 UTC m=+382.057530723" watchObservedRunningTime="2026-09-08 18:30:07.608126079 +0000 UTC m=+382.059701211"
Sep 08 18:30:18.587046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:18.587015    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8d688d88f-6ccqg"
Sep 08 18:30:31.755361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.755324    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"]
Sep 08 18:30:31.767396 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.767370    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.770736 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.770691    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 18:30:31.772999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.772097    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hsw52\""
Sep 08 18:30:31.772999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.772220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 18:30:31.772999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.772350    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 18:30:31.772999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.772931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 18:30:31.774072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.774012    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"]
Sep 08 18:30:31.774469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.774447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.774569 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.774486    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpw4\" (UniqueName: \"kubernetes.io/projected/caf80ae4-f7dd-44ec-ab5c-1613a234f114-kube-api-access-qvpw4\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.774983 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.774607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tmp\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.875542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.875502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.875769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.875551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpw4\" (UniqueName: \"kubernetes.io/projected/caf80ae4-f7dd-44ec-ab5c-1613a234f114-kube-api-access-qvpw4\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.875769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.875613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tmp\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.875769 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:31.875650    2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 08 18:30:31.875769 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:31.875757    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs podName:caf80ae4-f7dd-44ec-ab5c-1613a234f114 nodeName:}" failed. No retries permitted until 2026-09-08 18:30:32.375707822 +0000 UTC m=+406.827282940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs") pod "kube-auth-proxy-6d985cc74d-9hb2s" (UID: "caf80ae4-f7dd-44ec-ab5c-1613a234f114") : secret "kube-auth-proxy-tls" not found
Sep 08 18:30:31.878049 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.878022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tmp\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.889032 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.889011    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpw4\" (UniqueName: \"kubernetes.io/projected/caf80ae4-f7dd-44ec-ab5c-1613a234f114-kube-api-access-qvpw4\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:31.957167 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.957136    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rwjb"]
Sep 08 18:30:31.959779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.959752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:31.962500 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.962478    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7nrz8\""
Sep 08 18:30:31.962688 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.962668    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 08 18:30:31.970845 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.970585    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rwjb"]
Sep 08 18:30:31.976291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.976267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lslp\" (UniqueName: \"kubernetes.io/projected/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-kube-api-access-8lslp\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:31.976383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:31.976320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-cert\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.076768 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.076673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lslp\" (UniqueName: \"kubernetes.io/projected/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-kube-api-access-8lslp\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.076768 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.076718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-cert\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.079228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.079198    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-cert\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.085927 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.085902    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lslp\" (UniqueName: \"kubernetes.io/projected/29dc83f2-268a-4a5f-8f80-fa4d83b6141c-kube-api-access-8lslp\") pod \"odh-model-controller-7998c67dc4-9rwjb\" (UID: \"29dc83f2-268a-4a5f-8f80-fa4d83b6141c\") " pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.274559 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.274520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:32.378224 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.378149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:32.380570 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.380542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf80ae4-f7dd-44ec-ab5c-1613a234f114-tls-certs\") pod \"kube-auth-proxy-6d985cc74d-9hb2s\" (UID: \"caf80ae4-f7dd-44ec-ab5c-1613a234f114\") " pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:32.549299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.549264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9rwjb"]
Sep 08 18:30:32.553324 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:32.553296    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dc83f2_268a_4a5f_8f80_fa4d83b6141c.slice/crio-cec57f25df9b03f01cf79e79e3ef69e5e109e848fda104a35cdf8f2cff6a469e WatchSource:0}: Error finding container cec57f25df9b03f01cf79e79e3ef69e5e109e848fda104a35cdf8f2cff6a469e: Status 404 returned error can't find the container with id cec57f25df9b03f01cf79e79e3ef69e5e109e848fda104a35cdf8f2cff6a469e
Sep 08 18:30:32.642244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.642213    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kcz6"]
Sep 08 18:30:32.644830 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.644813    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.649701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.649682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 18:30:32.664575 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.664549    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kcz6"]
Sep 08 18:30:32.668630 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.668602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb" event={"ID":"29dc83f2-268a-4a5f-8f80-fa4d83b6141c","Type":"ContainerStarted","Data":"cec57f25df9b03f01cf79e79e3ef69e5e109e848fda104a35cdf8f2cff6a469e"}
Sep 08 18:30:32.678439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.678417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"
Sep 08 18:30:32.679712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.679695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.679798 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.679749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkg6k\" (UniqueName: \"kubernetes.io/projected/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-kube-api-access-tkg6k\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.780206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.780176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.780647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.780231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkg6k\" (UniqueName: \"kubernetes.io/projected/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-kube-api-access-tkg6k\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.780647 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:32.780372    2582 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 08 18:30:32.780647 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:32.780451    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs podName:5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c nodeName:}" failed. No retries permitted until 2026-09-08 18:30:33.280429471 +0000 UTC m=+407.732004598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs") pod "model-serving-api-54d89bd5fc-5kcz6" (UID: "5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c") : secret "model-serving-api-tls" not found
Sep 08 18:30:32.790334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.790308    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkg6k\" (UniqueName: \"kubernetes.io/projected/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-kube-api-access-tkg6k\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:32.951706 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:32.951680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s"]
Sep 08 18:30:32.953839 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:32.953812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf80ae4_f7dd_44ec_ab5c_1613a234f114.slice/crio-4b46b6f0677219c6c6f2b957a8035be36de3bdc07685da3ac8cba9eafdabd515 WatchSource:0}: Error finding container 4b46b6f0677219c6c6f2b957a8035be36de3bdc07685da3ac8cba9eafdabd515: Status 404 returned error can't find the container with id 4b46b6f0677219c6c6f2b957a8035be36de3bdc07685da3ac8cba9eafdabd515
Sep 08 18:30:33.132772 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.132743    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:33.135251 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.135235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.138519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.138491    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 18:30:33.138519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.138515    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 18:30:33.138871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.138855    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 18:30:33.139040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.139028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 18:30:33.139088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.139063    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8dbrq\""
Sep 08 18:30:33.148274 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.148250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:33.181790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.181892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181801    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.181892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.181892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.181892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb29v\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.182095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.181904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.182095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.182000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283372 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb29v\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283372 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:33.283585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.283766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.283742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.284170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.284119    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.286600 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.286572    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.286712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.286648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.286801 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.286714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.286880 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.286860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.286930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.286922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-5kcz6\" (UID: \"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:33.293181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.293135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb29v\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.293495 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.293471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-h96wf\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.444515 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.444475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:33.555099 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.555016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:33.675104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.675055    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s" event={"ID":"caf80ae4-f7dd-44ec-ab5c-1613a234f114","Type":"ContainerStarted","Data":"4b46b6f0677219c6c6f2b957a8035be36de3bdc07685da3ac8cba9eafdabd515"}
Sep 08 18:30:33.725282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.725255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:33.727805 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:33.727766    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3bc8f8_69a1_43e9_985a_2c47a97023e8.slice/crio-d468b77337a387de2fed32f0ec572397ce73d3026d9810e3ecd66365f0e083ee WatchSource:0}: Error finding container d468b77337a387de2fed32f0ec572397ce73d3026d9810e3ecd66365f0e083ee: Status 404 returned error can't find the container with id d468b77337a387de2fed32f0ec572397ce73d3026d9810e3ecd66365f0e083ee
Sep 08 18:30:33.836249 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:33.836109    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5kcz6"]
Sep 08 18:30:33.841399 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:33.841368    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d58d9bc_d281_4cae_a9b2_1cbd07f1d25c.slice/crio-38f609583050aba34d68ae049b3118c609c103a6a59c733da55ba7bff5e77871 WatchSource:0}: Error finding container 38f609583050aba34d68ae049b3118c609c103a6a59c733da55ba7bff5e77871: Status 404 returned error can't find the container with id 38f609583050aba34d68ae049b3118c609c103a6a59c733da55ba7bff5e77871
Sep 08 18:30:34.684797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:34.684755    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6" event={"ID":"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c","Type":"ContainerStarted","Data":"38f609583050aba34d68ae049b3118c609c103a6a59c733da55ba7bff5e77871"}
Sep 08 18:30:34.687025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:34.686977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" event={"ID":"3f3bc8f8-69a1-43e9-985a-2c47a97023e8","Type":"ContainerStarted","Data":"d468b77337a387de2fed32f0ec572397ce73d3026d9810e3ecd66365f0e083ee"}
Sep 08 18:30:37.701644 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.701604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s" event={"ID":"caf80ae4-f7dd-44ec-ab5c-1613a234f114","Type":"ContainerStarted","Data":"21f5be3af1cdc915c843f0046ac965cd51984afe4c8987c8807ed735075cc80e"}
Sep 08 18:30:37.703772 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.703686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb" event={"ID":"29dc83f2-268a-4a5f-8f80-fa4d83b6141c","Type":"ContainerStarted","Data":"0bdc1bb750c8700b1c38cfc5d190640ad7a145ffc70336ce400cdd034cabe255"}
Sep 08 18:30:37.704012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.703944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:37.705504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.705457    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6" event={"ID":"5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c","Type":"ContainerStarted","Data":"1f18bca73cb03d55d5357e7c70cf05bf90f80d03526100f76bc8d9d131237270"}
Sep 08 18:30:37.705616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.705594    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:30:37.736096 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.736034    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d985cc74d-9hb2s" podStartSLOduration=2.13135535 podStartE2EDuration="6.736013605s" totalImagesPullingTime="4.604658255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:37.734604028 +0000 UTC m=+412.186179161" watchObservedRunningTime="2026-09-08 18:30:37.736013605 +0000 UTC m=+412.187588737"
Sep 08 18:30:37.760565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.760499    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb" podStartSLOduration=1.754900226 podStartE2EDuration="6.760478636s" totalImagesPullingTime="5.00557841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:37.759044499 +0000 UTC m=+412.210619637" watchObservedRunningTime="2026-09-08 18:30:37.760478636 +0000 UTC m=+412.212053769"
Sep 08 18:30:37.788220 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:37.788122    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6" podStartSLOduration=2.041233435 podStartE2EDuration="5.788099389s" totalImagesPullingTime="3.746865954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:37.787593514 +0000 UTC m=+412.239168648" watchObservedRunningTime="2026-09-08 18:30:37.788099389 +0000 UTC m=+412.239674524"
Sep 08 18:30:38.880454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:38.880413    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:30:38.880766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:38.880506    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:30:39.714870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:39.714834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" event={"ID":"3f3bc8f8-69a1-43e9-985a-2c47a97023e8","Type":"ContainerStarted","Data":"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"}
Sep 08 18:30:39.715074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:39.715021    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:39.716675 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:39.716629    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-h96wf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 18:30:39.716817 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:39.716704    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:30:39.737478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:39.737431    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" podStartSLOduration=1.58768913 podStartE2EDuration="6.737416546s" totalImagesPullingTime="5.149727416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:39.735431486 +0000 UTC m=+414.187006632" watchObservedRunningTime="2026-09-08 18:30:39.737416546 +0000 UTC m=+414.188991678"
Sep 08 18:30:40.201196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.201161    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"]
Sep 08 18:30:40.203287 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.203267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.206674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.206654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5db2z\""
Sep 08 18:30:40.224520 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.224493    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"]
Sep 08 18:30:40.241861 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.241833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.241884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.241914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.241972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242200 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.242001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz76w\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-kube-api-access-zz76w\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242200 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.242031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242200 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.242060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242200 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.242098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.242200 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.242124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d5926a-58dc-4de1-859e-3cde04f4585c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.342960 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.342918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.342968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d5926a-58dc-4de1-859e-3cde04f4585c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343403 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz76w\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-kube-api-access-zz76w\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343403 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343403 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343747 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.343797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.343778    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53d5926a-58dc-4de1-859e-3cde04f4585c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.345887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.345865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.345997 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.345978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.358320 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.358300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.358714 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.358693    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz76w\" (UniqueName: \"kubernetes.io/projected/53d5926a-58dc-4de1-859e-3cde04f4585c-kube-api-access-zz76w\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x\" (UID: \"53d5926a-58dc-4de1-859e-3cde04f4585c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.514676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.514590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:40.720058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.720031    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:40.797498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:40.797414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"]
Sep 08 18:30:40.801462 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:40.801433    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d5926a_58dc_4de1_859e_3cde04f4585c.slice/crio-ba523cc0e6bda1c3983c06d7c35c51ada4834706d4e2833e8f0068a1fd2c1202 WatchSource:0}: Error finding container ba523cc0e6bda1c3983c06d7c35c51ada4834706d4e2833e8f0068a1fd2c1202: Status 404 returned error can't find the container with id ba523cc0e6bda1c3983c06d7c35c51ada4834706d4e2833e8f0068a1fd2c1202
Sep 08 18:30:41.724661 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:41.724613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x" event={"ID":"53d5926a-58dc-4de1-859e-3cde04f4585c","Type":"ContainerStarted","Data":"ba523cc0e6bda1c3983c06d7c35c51ada4834706d4e2833e8f0068a1fd2c1202"}
Sep 08 18:30:42.012071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.011958    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"]
Sep 08 18:30:42.014458 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.014429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.018027 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.017999    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 08 18:30:42.019215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.019193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qbzlv\""
Sep 08 18:30:42.021844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.020461    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"]
Sep 08 18:30:42.022970 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.022950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.026852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.026714    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7twfb\""
Sep 08 18:30:42.027097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.027086    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 08 18:30:42.028760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.028676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"]
Sep 08 18:30:42.038040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.038015    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"]
Sep 08 18:30:42.057330 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.057304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.057511 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.057359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lc6\" (UniqueName: \"kubernetes.io/projected/00bd7b67-4acc-41ae-bc51-4510fc333952-kube-api-access-b4lc6\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.057511 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.057408    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.057511 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.057438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgvb\" (UniqueName: \"kubernetes.io/projected/127b620a-72bc-4954-ae6a-b8fe38eb0b77-kube-api-access-wrgvb\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.158826 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.158769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.159012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.158845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lc6\" (UniqueName: \"kubernetes.io/projected/00bd7b67-4acc-41ae-bc51-4510fc333952-kube-api-access-b4lc6\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.159012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.158882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.159012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.158912    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgvb\" (UniqueName: \"kubernetes.io/projected/127b620a-72bc-4954-ae6a-b8fe38eb0b77-kube-api-access-wrgvb\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.159377 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:42.159218    2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 08 18:30:42.159377 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:42.159281    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert podName:00bd7b67-4acc-41ae-bc51-4510fc333952 nodeName:}" failed. No retries permitted until 2026-09-08 18:30:42.65926157 +0000 UTC m=+417.110836681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert") pod "kserve-controller-manager-d6f4b65b8-xctwd" (UID: "00bd7b67-4acc-41ae-bc51-4510fc333952") : secret "kserve-webhook-server-cert" not found
Sep 08 18:30:42.159377 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:42.159281    2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 08 18:30:42.159377 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:42.159344    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert podName:127b620a-72bc-4954-ae6a-b8fe38eb0b77 nodeName:}" failed. No retries permitted until 2026-09-08 18:30:42.659325772 +0000 UTC m=+417.110900886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert") pod "llmisvc-controller-manager-777c6bd7d5-s4j9j" (UID: "127b620a-72bc-4954-ae6a-b8fe38eb0b77") : secret "llmisvc-webhook-server-cert" not found
Sep 08 18:30:42.174263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.174239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lc6\" (UniqueName: \"kubernetes.io/projected/00bd7b67-4acc-41ae-bc51-4510fc333952-kube-api-access-b4lc6\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.175982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.175933    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgvb\" (UniqueName: \"kubernetes.io/projected/127b620a-72bc-4954-ae6a-b8fe38eb0b77-kube-api-access-wrgvb\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.662928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.662888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.663133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.662986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.665760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.665710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/127b620a-72bc-4954-ae6a-b8fe38eb0b77-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s4j9j\" (UID: \"127b620a-72bc-4954-ae6a-b8fe38eb0b77\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:42.665966 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.665937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/00bd7b67-4acc-41ae-bc51-4510fc333952-cert\") pod \"kserve-controller-manager-d6f4b65b8-xctwd\" (UID: \"00bd7b67-4acc-41ae-bc51-4510fc333952\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.932514 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.932413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:42.946774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:42.946750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:43.701759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:43.701708    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:30:43.701875 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:43.701813    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:30:43.701875 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:43.701843    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:30:43.945030 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:43.945005    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"]
Sep 08 18:30:43.948804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:43.948778    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"]
Sep 08 18:30:43.954134 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:30:43.954103    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod127b620a_72bc_4954_ae6a_b8fe38eb0b77.slice/crio-39fecd833cef5a43b86667a59fee1bbe3f26a2bb1d760c4bb42321443c5ddcfa WatchSource:0}: Error finding container 39fecd833cef5a43b86667a59fee1bbe3f26a2bb1d760c4bb42321443c5ddcfa: Status 404 returned error can't find the container with id 39fecd833cef5a43b86667a59fee1bbe3f26a2bb1d760c4bb42321443c5ddcfa
Sep 08 18:30:44.020697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.020660    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:44.020892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.020861    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerName="discovery" containerID="cri-o://f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223" gracePeriod=30
Sep 08 18:30:44.254282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.254259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:44.277205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277152    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277202    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb29v\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277346    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277379    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert\") pod \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\" (UID: \"3f3bc8f8-69a1-43e9-985a-2c47a97023e8\") "
Sep 08 18:30:44.277631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.277595    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:30:44.280191 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.280142    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts" (OuterVolumeSpecName: "cacerts") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:30:44.280400 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.280375    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:30:44.280513 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.280407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v" (OuterVolumeSpecName: "kube-api-access-gb29v") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "kube-api-access-gb29v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:30:44.280571 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.280547    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:30:44.280571 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.280554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:30:44.281296 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.281277    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs" (OuterVolumeSpecName: "local-certs") pod "3f3bc8f8-69a1-43e9-985a-2c47a97023e8" (UID: "3f3bc8f8-69a1-43e9-985a-2c47a97023e8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:30:44.378874 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378840    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-local-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.378874 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378868    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-kubeconfig\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.378874 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378877    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-dns-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.379110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378887    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-csr-ca-configmap\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.379110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378896    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.379110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378904    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-cacerts\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.379110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.378911    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb29v\" (UniqueName: \"kubernetes.io/projected/3f3bc8f8-69a1-43e9-985a-2c47a97023e8-kube-api-access-gb29v\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:30:44.742487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.742450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x" event={"ID":"53d5926a-58dc-4de1-859e-3cde04f4585c","Type":"ContainerStarted","Data":"4bc8d1a904569ec79a67d831318863ae000737727d0554bc5b254916a6492447"}
Sep 08 18:30:44.743776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.743749    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerID="f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223" exitCode=0
Sep 08 18:30:44.743892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.743783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"
Sep 08 18:30:44.743892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.743830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" event={"ID":"3f3bc8f8-69a1-43e9-985a-2c47a97023e8","Type":"ContainerDied","Data":"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"}
Sep 08 18:30:44.743892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.743882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf" event={"ID":"3f3bc8f8-69a1-43e9-985a-2c47a97023e8","Type":"ContainerDied","Data":"d468b77337a387de2fed32f0ec572397ce73d3026d9810e3ecd66365f0e083ee"}
Sep 08 18:30:44.744058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.743909    2582 scope.go:117] "RemoveContainer" containerID="f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"
Sep 08 18:30:44.745280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.745257    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j" event={"ID":"127b620a-72bc-4954-ae6a-b8fe38eb0b77","Type":"ContainerStarted","Data":"39fecd833cef5a43b86667a59fee1bbe3f26a2bb1d760c4bb42321443c5ddcfa"}
Sep 08 18:30:44.746645 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.746621    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd" event={"ID":"00bd7b67-4acc-41ae-bc51-4510fc333952","Type":"ContainerStarted","Data":"3931bbcec22c7f174f0e395dd46dc51ba05163759f1a19154adcc5b5fb8ea42f"}
Sep 08 18:30:44.752989 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.752958    2582 scope.go:117] "RemoveContainer" containerID="f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"
Sep 08 18:30:44.753263 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:30:44.753244    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223\": container with ID starting with f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223 not found: ID does not exist" containerID="f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"
Sep 08 18:30:44.753321 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.753272    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223"} err="failed to get container status \"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223\": rpc error: code = NotFound desc = could not find container \"f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223\": container with ID starting with f8c6111d04d3d98865c225faa35fe5898ecabf7bd3a9e6b29083e4601da79223 not found: ID does not exist"
Sep 08 18:30:44.767524 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.767485    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x" podStartSLOduration=1.86973592 podStartE2EDuration="4.767471069s" totalImagesPullingTime="2.897735149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:44.765697375 +0000 UTC m=+419.217272506" watchObservedRunningTime="2026-09-08 18:30:44.767471069 +0000 UTC m=+419.219046201"
Sep 08 18:30:44.780371 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.780345    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:44.786610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:44.786590    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-h96wf"]
Sep 08 18:30:45.515583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:45.515532    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:45.520969 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:45.520948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:45.752935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:45.752905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:45.753949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:45.753928    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x"
Sep 08 18:30:46.104900 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:46.104854    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" path="/var/lib/kubelet/pods/3f3bc8f8-69a1-43e9-985a-2c47a97023e8/volumes"
Sep 08 18:30:46.759917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:46.759876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd" event={"ID":"00bd7b67-4acc-41ae-bc51-4510fc333952","Type":"ContainerStarted","Data":"28e7997d1841e8d14469940f4fcab1e13a868653d839432ead8bf3f0946e4ae5"}
Sep 08 18:30:46.760360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:46.760050    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:30:46.778457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:46.778403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd" podStartSLOduration=3.084137589 podStartE2EDuration="5.77838542s" totalImagesPullingTime="2.694247831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:46.777453257 +0000 UTC m=+421.229028392" watchObservedRunningTime="2026-09-08 18:30:46.77838542 +0000 UTC m=+421.229960551"
Sep 08 18:30:47.766401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:47.766366    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j" event={"ID":"127b620a-72bc-4954-ae6a-b8fe38eb0b77","Type":"ContainerStarted","Data":"f9d7da89506251cf99ab4d04bda1fb936167d6a497bf715b6166f044393d7d30"}
Sep 08 18:30:47.766828 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:47.766529    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:30:47.784523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:47.784473    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j" podStartSLOduration=2.501682235 podStartE2EDuration="5.784456227s" totalImagesPullingTime="3.282773992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:30:47.783502266 +0000 UTC m=+422.235077394" watchObservedRunningTime="2026-09-08 18:30:47.784456227 +0000 UTC m=+422.236031358"
Sep 08 18:30:48.711848 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:48.711820    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9rwjb"
Sep 08 18:30:48.713985 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:30:48.713967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5kcz6"
Sep 08 18:31:17.771531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:17.771501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xctwd"
Sep 08 18:31:18.772792 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:18.772758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s4j9j"
Sep 08 18:31:48.505936 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.505902    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"]
Sep 08 18:31:48.506497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.506303    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerName="discovery"
Sep 08 18:31:48.506497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.506319    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerName="discovery"
Sep 08 18:31:48.506497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.506410    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3bc8f8-69a1-43e9-985a-2c47a97023e8" containerName="discovery"
Sep 08 18:31:48.508563 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.508543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.512051 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.512028    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 18:31:48.512424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.512408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 18:31:48.512540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.512521    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9ld8f\""
Sep 08 18:31:48.542479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.542447    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"]
Sep 08 18:31:48.582458 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.582419    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.582605 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.582474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcwl\" (UniqueName: \"kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.683829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.683793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.683994 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.683852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcwl\" (UniqueName: \"kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.684166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.684146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.693646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.693621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcwl\" (UniqueName: \"kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:48.818361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:48.818259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:49.098851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:49.098825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"]
Sep 08 18:31:49.101701 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:31:49.101672    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00256b3e_65b5_4d44_a26c_f37e3938727e.slice/crio-10bd70e46fc128a6db0455dd272c72d529d1c608c3ed3045edcc0987b87137ee WatchSource:0}: Error finding container 10bd70e46fc128a6db0455dd272c72d529d1c608c3ed3045edcc0987b87137ee: Status 404 returned error can't find the container with id 10bd70e46fc128a6db0455dd272c72d529d1c608c3ed3045edcc0987b87137ee
Sep 08 18:31:50.019815 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:50.019714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc" event={"ID":"00256b3e-65b5-4d44-a26c-f37e3938727e","Type":"ContainerStarted","Data":"10bd70e46fc128a6db0455dd272c72d529d1c608c3ed3045edcc0987b87137ee"}
Sep 08 18:31:50.889181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:50.889142    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ltp7b"]
Sep 08 18:31:50.895499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:50.895471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:50.898657 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:50.898627    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zgl4v\""
Sep 08 18:31:50.908886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:50.908818    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ltp7b"]
Sep 08 18:31:51.003647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:51.003606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkr2f\" (UniqueName: \"kubernetes.io/projected/e7c13d54-944a-463e-aa56-98dc106c4d8d-kube-api-access-dkr2f\") pod \"authorino-operator-76b478bd78-ltp7b\" (UID: \"e7c13d54-944a-463e-aa56-98dc106c4d8d\") " pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:51.104809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:51.104769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr2f\" (UniqueName: \"kubernetes.io/projected/e7c13d54-944a-463e-aa56-98dc106c4d8d-kube-api-access-dkr2f\") pod \"authorino-operator-76b478bd78-ltp7b\" (UID: \"e7c13d54-944a-463e-aa56-98dc106c4d8d\") " pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:51.115067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:51.115038    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr2f\" (UniqueName: \"kubernetes.io/projected/e7c13d54-944a-463e-aa56-98dc106c4d8d-kube-api-access-dkr2f\") pod \"authorino-operator-76b478bd78-ltp7b\" (UID: \"e7c13d54-944a-463e-aa56-98dc106c4d8d\") " pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:51.210236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:51.210153    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:51.710881 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:51.710846    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-ltp7b"]
Sep 08 18:31:51.714902 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:31:51.714864    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c13d54_944a_463e_aa56_98dc106c4d8d.slice/crio-87f089b35572de44f1c3c477f3a214aa866649e03ba3098cc2ddad9bafe0197c WatchSource:0}: Error finding container 87f089b35572de44f1c3c477f3a214aa866649e03ba3098cc2ddad9bafe0197c: Status 404 returned error can't find the container with id 87f089b35572de44f1c3c477f3a214aa866649e03ba3098cc2ddad9bafe0197c
Sep 08 18:31:52.031404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:52.031359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b" event={"ID":"e7c13d54-944a-463e-aa56-98dc106c4d8d","Type":"ContainerStarted","Data":"87f089b35572de44f1c3c477f3a214aa866649e03ba3098cc2ddad9bafe0197c"}
Sep 08 18:31:53.162567 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.162538    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"]
Sep 08 18:31:53.165113 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.165093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:53.168175 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.168065    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zv7x8\""
Sep 08 18:31:53.178792 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.178742    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"]
Sep 08 18:31:53.223223 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.223187    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jh2k\" (UniqueName: \"kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k\") pod \"limitador-operator-controller-manager-74495c69f8-fbppq\" (UID: \"369f536a-5e82-4cda-a465-7e5927fba8d3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:53.323917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.323874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jh2k\" (UniqueName: \"kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k\") pod \"limitador-operator-controller-manager-74495c69f8-fbppq\" (UID: \"369f536a-5e82-4cda-a465-7e5927fba8d3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:53.332993 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.332969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jh2k\" (UniqueName: \"kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k\") pod \"limitador-operator-controller-manager-74495c69f8-fbppq\" (UID: \"369f536a-5e82-4cda-a465-7e5927fba8d3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:53.478486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:53.478438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:54.004543 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.004518    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"]
Sep 08 18:31:54.006766 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:31:54.006718    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369f536a_5e82_4cda_a465_7e5927fba8d3.slice/crio-f5c0d17a8e5ef01d74285909c026570ccd66cb0acf7337ee7c45ee6c61a8c0f4 WatchSource:0}: Error finding container f5c0d17a8e5ef01d74285909c026570ccd66cb0acf7337ee7c45ee6c61a8c0f4: Status 404 returned error can't find the container with id f5c0d17a8e5ef01d74285909c026570ccd66cb0acf7337ee7c45ee6c61a8c0f4
Sep 08 18:31:54.043107 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.043071    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc" event={"ID":"00256b3e-65b5-4d44-a26c-f37e3938727e","Type":"ContainerStarted","Data":"b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe"}
Sep 08 18:31:54.043255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.043196    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:31:54.044926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.044903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b" event={"ID":"e7c13d54-944a-463e-aa56-98dc106c4d8d","Type":"ContainerStarted","Data":"21c9dfe816241e433c3cc76475322ae74efd4376e0f02e214b66404e23894db2"}
Sep 08 18:31:54.045030 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.044962    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:31:54.046190 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.046170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq" event={"ID":"369f536a-5e82-4cda-a465-7e5927fba8d3","Type":"ContainerStarted","Data":"f5c0d17a8e5ef01d74285909c026570ccd66cb0acf7337ee7c45ee6c61a8c0f4"}
Sep 08 18:31:54.069490 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.069441    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc" podStartSLOduration=2.063930401 podStartE2EDuration="6.069426198s" totalImagesPullingTime="4.005495797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:31:54.067421851 +0000 UTC m=+488.518996985" watchObservedRunningTime="2026-09-08 18:31:54.069426198 +0000 UTC m=+488.521001329"
Sep 08 18:31:54.089932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:54.089892    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b" podStartSLOduration=2.038567529 podStartE2EDuration="4.089876609s" totalImagesPullingTime="2.05130908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:31:54.089142262 +0000 UTC m=+488.540717395" watchObservedRunningTime="2026-09-08 18:31:54.089876609 +0000 UTC m=+488.541451741"
Sep 08 18:31:56.058111 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:56.058025    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq" event={"ID":"369f536a-5e82-4cda-a465-7e5927fba8d3","Type":"ContainerStarted","Data":"44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4"}
Sep 08 18:31:56.058464 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:56.058154    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:31:56.076305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:31:56.076244    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq" podStartSLOduration=1.367846558 podStartE2EDuration="3.076226962s" totalImagesPullingTime="1.708380404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:31:56.075647107 +0000 UTC m=+490.527222240" watchObservedRunningTime="2026-09-08 18:31:56.076226962 +0000 UTC m=+490.527802093"
Sep 08 18:32:05.053791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:05.053758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:05.054229 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:05.053836    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-ltp7b"
Sep 08 18:32:07.064777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.064749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.184532 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.184497    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"]
Sep 08 18:32:07.184772 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.184742    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" containerName="manager" containerID="cri-o://b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe" gracePeriod=2
Sep 08 18:32:07.202584 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.202559    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"]
Sep 08 18:32:07.206991 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.206961    2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"00256b3e-65b5-4d44-a26c-f37e3938727e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\": pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.222315 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.222291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:07.222629 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.222614    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" containerName="manager"
Sep 08 18:32:07.222684 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.222630    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" containerName="manager"
Sep 08 18:32:07.222735 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.222709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" containerName="manager"
Sep 08 18:32:07.224664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.224646    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"]
Sep 08 18:32:07.224832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.224809    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" containerName="manager" containerID="cri-o://44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4" gracePeriod=2
Sep 08 18:32:07.224922 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.224901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.242398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.242367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:07.244156 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.244130    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"]
Sep 08 18:32:07.253042 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.253005    2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"369f536a-5e82-4cda-a465-7e5927fba8d3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-fbppq\": pods \"limitador-operator-controller-manager-74495c69f8-fbppq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.256701 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.256672    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.264316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.264295    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"]
Sep 08 18:32:07.264619 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.264599    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" containerName="manager"
Sep 08 18:32:07.264669 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.264621    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" containerName="manager"
Sep 08 18:32:07.264705 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.264683    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" containerName="manager"
Sep 08 18:32:07.266477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.266461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:07.281914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.281890    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"]
Sep 08 18:32:07.285801 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.285767    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fbppq\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.288198 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.288166    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.290678 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.290647    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fbppq\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.292985 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.292952    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.334184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.334152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskrw\" (UniqueName: \"kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.334310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.334199    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.334310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.334286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4vq\" (UniqueName: \"kubernetes.io/projected/d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30-kube-api-access-lj4vq\") pod \"limitador-operator-controller-manager-74495c69f8-snrsq\" (UID: \"d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:07.435278 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.435243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4vq\" (UniqueName: \"kubernetes.io/projected/d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30-kube-api-access-lj4vq\") pod \"limitador-operator-controller-manager-74495c69f8-snrsq\" (UID: \"d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:07.435457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.435313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskrw\" (UniqueName: \"kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.435457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.435336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.435790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.435741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.445228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.445197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4vq\" (UniqueName: \"kubernetes.io/projected/d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30-kube-api-access-lj4vq\") pod \"limitador-operator-controller-manager-74495c69f8-snrsq\" (UID: \"d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:07.445440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.445419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskrw\" (UniqueName: \"kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sjljb\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.466930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.466911    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.467990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.467954    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.469636 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.469614    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fbppq\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.471837 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.471811    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.473924 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.473903    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fbppq\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:07.475939 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:07.475923    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-xfbfc\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:07.536095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.536072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jh2k\" (UniqueName: \"kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k\") pod \"369f536a-5e82-4cda-a465-7e5927fba8d3\" (UID: \"369f536a-5e82-4cda-a465-7e5927fba8d3\") "
Sep 08 18:32:07.536198 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.536107    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume\") pod \"00256b3e-65b5-4d44-a26c-f37e3938727e\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") "
Sep 08 18:32:07.536198 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.536125    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhcwl\" (UniqueName: \"kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl\") pod \"00256b3e-65b5-4d44-a26c-f37e3938727e\" (UID: \"00256b3e-65b5-4d44-a26c-f37e3938727e\") "
Sep 08 18:32:07.536621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.536587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "00256b3e-65b5-4d44-a26c-f37e3938727e" (UID: "00256b3e-65b5-4d44-a26c-f37e3938727e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:32:07.538382 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.538361    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl" (OuterVolumeSpecName: "kube-api-access-rhcwl") pod "00256b3e-65b5-4d44-a26c-f37e3938727e" (UID: "00256b3e-65b5-4d44-a26c-f37e3938727e"). InnerVolumeSpecName "kube-api-access-rhcwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:32:07.538456 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.538383    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k" (OuterVolumeSpecName: "kube-api-access-9jh2k") pod "369f536a-5e82-4cda-a465-7e5927fba8d3" (UID: "369f536a-5e82-4cda-a465-7e5927fba8d3"). InnerVolumeSpecName "kube-api-access-9jh2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:32:07.600637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.600558    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:07.606481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.606463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:07.637512 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.637480    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00256b3e-65b5-4d44-a26c-f37e3938727e-extensions-socket-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:07.637512 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.637509    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcwl\" (UniqueName: \"kubernetes.io/projected/00256b3e-65b5-4d44-a26c-f37e3938727e-kube-api-access-rhcwl\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:07.637512 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.637520    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jh2k\" (UniqueName: \"kubernetes.io/projected/369f536a-5e82-4cda-a465-7e5927fba8d3-kube-api-access-9jh2k\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:07.885092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.884950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:07.887166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:07.887137    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"]
Sep 08 18:32:07.887605 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:32:07.887585    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3063d99f_c6e5_47fa_a5bc_7bc6c9dab131.slice/crio-bc2fdf20782433687bd686196385f11f67cb75e5b7df7c1d1a80ab71ba382122 WatchSource:0}: Error finding container bc2fdf20782433687bd686196385f11f67cb75e5b7df7c1d1a80ab71ba382122: Status 404 returned error can't find the container with id bc2fdf20782433687bd686196385f11f67cb75e5b7df7c1d1a80ab71ba382122
Sep 08 18:32:07.890199 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:32:07.890177    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd818f92f_c8f3_4c2c_9f2e_d8c1c2a79f30.slice/crio-6d5d983f1dbb522b00a34144b41f92c1df3bd0c9ab5ec27eeb97cb32603d7575 WatchSource:0}: Error finding container 6d5d983f1dbb522b00a34144b41f92c1df3bd0c9ab5ec27eeb97cb32603d7575: Status 404 returned error can't find the container with id 6d5d983f1dbb522b00a34144b41f92c1df3bd0c9ab5ec27eeb97cb32603d7575
Sep 08 18:32:08.099371 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.099339    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00256b3e-65b5-4d44-a26c-f37e3938727e" path="/var/lib/kubelet/pods/00256b3e-65b5-4d44-a26c-f37e3938727e/volumes"
Sep 08 18:32:08.099719 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.099645    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="369f536a-5e82-4cda-a465-7e5927fba8d3" path="/var/lib/kubelet/pods/369f536a-5e82-4cda-a465-7e5927fba8d3/volumes"
Sep 08 18:32:08.110265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.110242    2582 generic.go:356] "Generic (PLEG): container finished" podID="00256b3e-65b5-4d44-a26c-f37e3938727e" containerID="b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe" exitCode=0
Sep 08 18:32:08.110374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.110282    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-xfbfc"
Sep 08 18:32:08.110374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.110331    2582 scope.go:117] "RemoveContainer" containerID="b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe"
Sep 08 18:32:08.111673 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.111651    2582 generic.go:356] "Generic (PLEG): container finished" podID="369f536a-5e82-4cda-a465-7e5927fba8d3" containerID="44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4" exitCode=0
Sep 08 18:32:08.111799 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.111698    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fbppq"
Sep 08 18:32:08.113982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.113939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" event={"ID":"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131","Type":"ContainerStarted","Data":"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"}
Sep 08 18:32:08.114085 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.113995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" event={"ID":"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131","Type":"ContainerStarted","Data":"bc2fdf20782433687bd686196385f11f67cb75e5b7df7c1d1a80ab71ba382122"}
Sep 08 18:32:08.115677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.115655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq" event={"ID":"d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30","Type":"ContainerStarted","Data":"8b7d253c61df08e499b755b1a04465699dca3ba62812176892e992d7c021b0b8"}
Sep 08 18:32:08.115799 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.115700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq" event={"ID":"d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30","Type":"ContainerStarted","Data":"6d5d983f1dbb522b00a34144b41f92c1df3bd0c9ab5ec27eeb97cb32603d7575"}
Sep 08 18:32:08.115799 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.115774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:08.118989 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.118974    2582 scope.go:117] "RemoveContainer" containerID="b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe"
Sep 08 18:32:08.119215 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:08.119198    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe\": container with ID starting with b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe not found: ID does not exist" containerID="b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe"
Sep 08 18:32:08.119266 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.119222    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe"} err="failed to get container status \"b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe\": rpc error: code = NotFound desc = could not find container \"b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe\": container with ID starting with b396f479a3f86f4fcf080120b8655445d9a3a0ed945570c650af42ecdc9361fe not found: ID does not exist"
Sep 08 18:32:08.119266 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.119240    2582 scope.go:117] "RemoveContainer" containerID="44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4"
Sep 08 18:32:08.126518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.126497    2582 scope.go:117] "RemoveContainer" containerID="44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4"
Sep 08 18:32:08.126777 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:08.126761    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4\": container with ID starting with 44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4 not found: ID does not exist" containerID="44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4"
Sep 08 18:32:08.126834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.126783    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4"} err="failed to get container status \"44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4\": rpc error: code = NotFound desc = could not find container \"44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4\": container with ID starting with 44143b198e5d337f4cb3429acb804eaa54fc74c02ab54dcdc8e2d2c05a6a7ea4 not found: ID does not exist"
Sep 08 18:32:08.142806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.142766    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" podStartSLOduration=1.142754227 podStartE2EDuration="1.142754227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:08.140287013 +0000 UTC m=+502.591862146" watchObservedRunningTime="2026-09-08 18:32:08.142754227 +0000 UTC m=+502.594329356"
Sep 08 18:32:08.158327 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:08.158285    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq" podStartSLOduration=1.158270941 podStartE2EDuration="1.158270941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:08.157774127 +0000 UTC m=+502.609349260" watchObservedRunningTime="2026-09-08 18:32:08.158270941 +0000 UTC m=+502.609846073"
Sep 08 18:32:09.121998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:09.121967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:19.124333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:19.124303    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-snrsq"
Sep 08 18:32:20.129086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:20.129051    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:23.419298 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.419261    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:23.419691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.419435    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" podUID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" containerName="manager" containerID="cri-o://2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694" gracePeriod=10
Sep 08 18:32:23.668055 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.668028    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:23.755576 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.755493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kskrw\" (UniqueName: \"kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw\") pod \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") "
Sep 08 18:32:23.755576 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.755563    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume\") pod \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\" (UID: \"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131\") "
Sep 08 18:32:23.755992 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.755960    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" (UID: "3063d99f-c6e5-47fa-a5bc-7bc6c9dab131"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:32:23.757779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.757755    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw" (OuterVolumeSpecName: "kube-api-access-kskrw") pod "3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" (UID: "3063d99f-c6e5-47fa-a5bc-7bc6c9dab131"). InnerVolumeSpecName "kube-api-access-kskrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:32:23.856590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.856550    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskrw\" (UniqueName: \"kubernetes.io/projected/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-kube-api-access-kskrw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:23.856590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:23.856583    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131-extensions-socket-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:24.186370 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.186333    2582 generic.go:356] "Generic (PLEG): container finished" podID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" containerID="2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694" exitCode=0
Sep 08 18:32:24.186542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.186377    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"
Sep 08 18:32:24.186542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.186422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" event={"ID":"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131","Type":"ContainerDied","Data":"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"}
Sep 08 18:32:24.186542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.186474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb" event={"ID":"3063d99f-c6e5-47fa-a5bc-7bc6c9dab131","Type":"ContainerDied","Data":"bc2fdf20782433687bd686196385f11f67cb75e5b7df7c1d1a80ab71ba382122"}
Sep 08 18:32:24.186542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.186497    2582 scope.go:117] "RemoveContainer" containerID="2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"
Sep 08 18:32:24.195265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.195249    2582 scope.go:117] "RemoveContainer" containerID="2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"
Sep 08 18:32:24.195542 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:24.195522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694\": container with ID starting with 2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694 not found: ID does not exist" containerID="2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"
Sep 08 18:32:24.195595 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.195548    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694"} err="failed to get container status \"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694\": rpc error: code = NotFound desc = could not find container \"2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694\": container with ID starting with 2ca0de0af227fecf17c0cc631fa4b6a2ea8e118120ad2515322ead86d2b3b694 not found: ID does not exist"
Sep 08 18:32:24.211433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.211408    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:24.218355 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:24.218332    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sjljb"]
Sep 08 18:32:26.098312 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:26.098278    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" path="/var/lib/kubelet/pods/3063d99f-c6e5-47fa-a5bc-7bc6c9dab131/volumes"
Sep 08 18:32:39.729289 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.729195    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"]
Sep 08 18:32:39.729751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.729653    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" containerName="manager"
Sep 08 18:32:39.729751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.729671    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" containerName="manager"
Sep 08 18:32:39.729880 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.729767    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3063d99f-c6e5-47fa-a5bc-7bc6c9dab131" containerName="manager"
Sep 08 18:32:39.731593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.731572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.734354 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.734332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mxks5\""
Sep 08 18:32:39.756261 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.756233    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"]
Sep 08 18:32:39.889995 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.889954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245faeeb-9478-415a-b635-cfbee2b8cdd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbn4\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-kube-api-access-4dbn4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.890359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.890324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.990823 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.990823 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.990823 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbn4\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-kube-api-access-4dbn4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.990965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245faeeb-9478-415a-b635-cfbee2b8cdd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.991242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.991266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.991300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.991379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.991748 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.991703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245faeeb-9478-415a-b635-cfbee2b8cdd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.993538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.993515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.993647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.993616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.999423 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.999402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:39.999617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:39.999597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbn4\" (UniqueName: \"kubernetes.io/projected/245faeeb-9478-415a-b635-cfbee2b8cdd2-kube-api-access-4dbn4\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tmgsr\" (UID: \"245faeeb-9478-415a-b635-cfbee2b8cdd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:40.042623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:40.042579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:40.318203 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:40.318180    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"]
Sep 08 18:32:40.323997 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:40.323955    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:32:40.324097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:40.324047    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:32:40.324097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:40.324088    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:32:41.264251 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:41.264216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" event={"ID":"245faeeb-9478-415a-b635-cfbee2b8cdd2","Type":"ContainerStarted","Data":"71d6818eaed2f65fbeff805765b71c8434e92790a7f59f9b162e739184c95720"}
Sep 08 18:32:41.264790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:41.264269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" event={"ID":"245faeeb-9478-415a-b635-cfbee2b8cdd2","Type":"ContainerStarted","Data":"d011271c667378129e9fb3385002fcd9244a208273e00b63ca7365c6469d9cc5"}
Sep 08 18:32:41.286434 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:41.286380    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podStartSLOduration=2.286364428 podStartE2EDuration="2.286364428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:41.284812924 +0000 UTC m=+535.736388068" watchObservedRunningTime="2026-09-08 18:32:41.286364428 +0000 UTC m=+535.737939560"
Sep 08 18:32:42.043486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.043448    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:42.048079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.048056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:42.268797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.268762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:42.269683 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.269660    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:32:42.314415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.314334    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:42.316642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.316608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.320196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.320176    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9k5mn\""
Sep 08 18:32:42.320673 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.320650    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 18:32:42.328988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.328967    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:42.400143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.400105    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:42.411964 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.411928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.412149 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.412127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hchws\" (UniqueName: \"kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.513284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.513243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hchws\" (UniqueName: \"kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.513478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.513318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.514147 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.514122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.522262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.522236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hchws\" (UniqueName: \"kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws\") pod \"limitador-limitador-8588749977-2t7mj\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") " pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.630161 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.630053    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:42.901044 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:42.901018    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:42.903145 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:32:42.903106    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3b6d19_862f_44f6_acd4_2dcd9ce3d98f.slice/crio-7b9aa7deddeb181c2a24d2a9fb23f0f8b5a5ae8c22c99ee15df766b0629f17b7 WatchSource:0}: Error finding container 7b9aa7deddeb181c2a24d2a9fb23f0f8b5a5ae8c22c99ee15df766b0629f17b7: Status 404 returned error can't find the container with id 7b9aa7deddeb181c2a24d2a9fb23f0f8b5a5ae8c22c99ee15df766b0629f17b7
Sep 08 18:32:43.120850 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.120821    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:43.123402 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.123386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:43.126286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.126270    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lvhv4\""
Sep 08 18:32:43.132285 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.132264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:43.219438 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.219349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkhv\" (UniqueName: \"kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv\") pod \"authorino-7bcfd8f9cd-px88m\" (UID: \"c6e4c66f-065f-4a0b-a7dd-5fef33170707\") " pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:43.275019 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.274979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" event={"ID":"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f","Type":"ContainerStarted","Data":"7b9aa7deddeb181c2a24d2a9fb23f0f8b5a5ae8c22c99ee15df766b0629f17b7"}
Sep 08 18:32:43.319864 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.319828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkhv\" (UniqueName: \"kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv\") pod \"authorino-7bcfd8f9cd-px88m\" (UID: \"c6e4c66f-065f-4a0b-a7dd-5fef33170707\") " pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:43.328337 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.328305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkhv\" (UniqueName: \"kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv\") pod \"authorino-7bcfd8f9cd-px88m\" (UID: \"c6e4c66f-065f-4a0b-a7dd-5fef33170707\") " pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:43.433928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.433889    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:43.699217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:43.699161    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:43.701495 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:32:43.701469    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e4c66f_065f_4a0b_a7dd_5fef33170707.slice/crio-c500103fda3a792cb0e0a43124db7f7335885d25a505faa063577027db656e67 WatchSource:0}: Error finding container c500103fda3a792cb0e0a43124db7f7335885d25a505faa063577027db656e67: Status 404 returned error can't find the container with id c500103fda3a792cb0e0a43124db7f7335885d25a505faa063577027db656e67
Sep 08 18:32:44.282821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:44.282782    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" event={"ID":"c6e4c66f-065f-4a0b-a7dd-5fef33170707","Type":"ContainerStarted","Data":"c500103fda3a792cb0e0a43124db7f7335885d25a505faa063577027db656e67"}
Sep 08 18:32:45.288714 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:45.288681    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" event={"ID":"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f","Type":"ContainerStarted","Data":"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"}
Sep 08 18:32:45.289163 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:45.288779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:45.305362 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:45.305315    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" podStartSLOduration=1.6514861060000001 podStartE2EDuration="3.305298703s" totalImagesPullingTime="1.653812597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:45.304236164 +0000 UTC m=+539.755811502" watchObservedRunningTime="2026-09-08 18:32:45.305298703 +0000 UTC m=+539.756873849"
Sep 08 18:32:47.055861 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:47.055829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:49.309000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.308909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" event={"ID":"c6e4c66f-065f-4a0b-a7dd-5fef33170707","Type":"ContainerStarted","Data":"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"}
Sep 08 18:32:49.309465 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.309002    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" podUID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" containerName="authorino" containerID="cri-o://56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f" gracePeriod=30
Sep 08 18:32:49.324245 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.324202    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" podStartSLOduration=1.121086838 podStartE2EDuration="6.324187491s" totalImagesPullingTime="5.203100653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:32:49.323284312 +0000 UTC m=+543.774859447" watchObservedRunningTime="2026-09-08 18:32:49.324187491 +0000 UTC m=+543.775762622"
Sep 08 18:32:49.552230 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.552205    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:49.579265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.579203    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gkhv\" (UniqueName: \"kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv\") pod \"c6e4c66f-065f-4a0b-a7dd-5fef33170707\" (UID: \"c6e4c66f-065f-4a0b-a7dd-5fef33170707\") "
Sep 08 18:32:49.581306 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.581283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv" (OuterVolumeSpecName: "kube-api-access-8gkhv") pod "c6e4c66f-065f-4a0b-a7dd-5fef33170707" (UID: "c6e4c66f-065f-4a0b-a7dd-5fef33170707"). InnerVolumeSpecName "kube-api-access-8gkhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:32:49.680621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:49.680580    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkhv\" (UniqueName: \"kubernetes.io/projected/c6e4c66f-065f-4a0b-a7dd-5fef33170707-kube-api-access-8gkhv\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:50.314073 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.314038    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" containerID="56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f" exitCode=0
Sep 08 18:32:50.314463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.314082    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m"
Sep 08 18:32:50.314463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.314072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" event={"ID":"c6e4c66f-065f-4a0b-a7dd-5fef33170707","Type":"ContainerDied","Data":"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"}
Sep 08 18:32:50.314463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.314181    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-px88m" event={"ID":"c6e4c66f-065f-4a0b-a7dd-5fef33170707","Type":"ContainerDied","Data":"c500103fda3a792cb0e0a43124db7f7335885d25a505faa063577027db656e67"}
Sep 08 18:32:50.314463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.314199    2582 scope.go:117] "RemoveContainer" containerID="56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"
Sep 08 18:32:50.322233 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.322213    2582 scope.go:117] "RemoveContainer" containerID="56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"
Sep 08 18:32:50.322522 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:50.322505    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f\": container with ID starting with 56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f not found: ID does not exist" containerID="56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"
Sep 08 18:32:50.322581 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.322530    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f"} err="failed to get container status \"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f\": rpc error: code = NotFound desc = could not find container \"56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f\": container with ID starting with 56e4941d957c485a92d2dd8183e915102e8c1ce5407dbacb715577af08ea1d8f not found: ID does not exist"
Sep 08 18:32:50.334431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.334409    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:50.338629 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:50.338610    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-px88m"]
Sep 08 18:32:52.098085 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:52.098054    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" path="/var/lib/kubelet/pods/c6e4c66f-065f-4a0b-a7dd-5fef33170707/volumes"
Sep 08 18:32:56.294022 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:56.293990    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:58.055638 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.055601    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:58.056115 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.055804    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" podUID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" containerName="limitador" containerID="cri-o://4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3" gracePeriod=30
Sep 08 18:32:58.610646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.610621    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:58.649732 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.649700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file\") pod \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") "
Sep 08 18:32:58.649901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.649824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hchws\" (UniqueName: \"kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws\") pod \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\" (UID: \"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f\") "
Sep 08 18:32:58.650104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.650082    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file" (OuterVolumeSpecName: "config-file") pod "0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" (UID: "0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:32:58.651934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.651909    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws" (OuterVolumeSpecName: "kube-api-access-hchws") pod "0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" (UID: "0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f"). InnerVolumeSpecName "kube-api-access-hchws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:32:58.751226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.751193    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hchws\" (UniqueName: \"kubernetes.io/projected/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-kube-api-access-hchws\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:58.751226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:58.751221    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f-config-file\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:32:59.355008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.354973    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" containerID="4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3" exitCode=0
Sep 08 18:32:59.355435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.355013    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj"
Sep 08 18:32:59.355435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.355067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" event={"ID":"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f","Type":"ContainerDied","Data":"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"}
Sep 08 18:32:59.355435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.355122    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2t7mj" event={"ID":"0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f","Type":"ContainerDied","Data":"7b9aa7deddeb181c2a24d2a9fb23f0f8b5a5ae8c22c99ee15df766b0629f17b7"}
Sep 08 18:32:59.355435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.355150    2582 scope.go:117] "RemoveContainer" containerID="4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"
Sep 08 18:32:59.363920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.363892    2582 scope.go:117] "RemoveContainer" containerID="4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"
Sep 08 18:32:59.364294 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:32:59.364255    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3\": container with ID starting with 4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3 not found: ID does not exist" containerID="4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"
Sep 08 18:32:59.364380 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.364307    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3"} err="failed to get container status \"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3\": rpc error: code = NotFound desc = could not find container \"4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3\": container with ID starting with 4022eb220e7394d5a2c1c11c29f0f90116e0fffdd7b6b637a35653b92b55a3f3 not found: ID does not exist"
Sep 08 18:32:59.380274 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.380246    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:32:59.386196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:32:59.386173    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2t7mj"]
Sep 08 18:33:00.098549 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:00.098506    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" path="/var/lib/kubelet/pods/0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f/volumes"
Sep 08 18:33:03.970904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.970873    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"]
Sep 08 18:33:03.971457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971434    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" containerName="limitador"
Sep 08 18:33:03.971537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" containerName="limitador"
Sep 08 18:33:03.971537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971476    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" containerName="authorino"
Sep 08 18:33:03.971537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971484    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" containerName="authorino"
Sep 08 18:33:03.971683 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971564    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3b6d19-862f-44f6-acd4-2dcd9ce3d98f" containerName="limitador"
Sep 08 18:33:03.971683 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.971577    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e4c66f-065f-4a0b-a7dd-5fef33170707" containerName="authorino"
Sep 08 18:33:03.975835 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.975811    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:03.978821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.978796    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 18:33:03.978821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.978808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 18:33:03.980154 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.980131    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nnjrt\""
Sep 08 18:33:03.980265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.980133    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 08 18:33:03.983256 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:03.983062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"]
Sep 08 18:33:04.094375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.094334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fa33bc29-0138-4074-b22d-4b34e3b694c5-data\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.094542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.094397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvcd7\" (UniqueName: \"kubernetes.io/projected/fa33bc29-0138-4074-b22d-4b34e3b694c5-kube-api-access-tvcd7\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.195181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.195150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvcd7\" (UniqueName: \"kubernetes.io/projected/fa33bc29-0138-4074-b22d-4b34e3b694c5-kube-api-access-tvcd7\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.195385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.195245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fa33bc29-0138-4074-b22d-4b34e3b694c5-data\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.195622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.195600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fa33bc29-0138-4074-b22d-4b34e3b694c5-data\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.204001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.203970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvcd7\" (UniqueName: \"kubernetes.io/projected/fa33bc29-0138-4074-b22d-4b34e3b694c5-kube-api-access-tvcd7\") pod \"postgres-69fb4594cf-2h7f4\" (UID: \"fa33bc29-0138-4074-b22d-4b34e3b694c5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.287032 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.286949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:04.556280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:04.556210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"]
Sep 08 18:33:04.559876 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:33:04.559848    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa33bc29_0138_4074_b22d_4b34e3b694c5.slice/crio-94ab2b6fd9c59a5075b59243d8632e0a848787e83b2afbacc2c42b92f64e163e WatchSource:0}: Error finding container 94ab2b6fd9c59a5075b59243d8632e0a848787e83b2afbacc2c42b92f64e163e: Status 404 returned error can't find the container with id 94ab2b6fd9c59a5075b59243d8632e0a848787e83b2afbacc2c42b92f64e163e
Sep 08 18:33:05.385135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:05.385094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4" event={"ID":"fa33bc29-0138-4074-b22d-4b34e3b694c5","Type":"ContainerStarted","Data":"94ab2b6fd9c59a5075b59243d8632e0a848787e83b2afbacc2c42b92f64e163e"}
Sep 08 18:33:09.406733 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:09.406694    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4" event={"ID":"fa33bc29-0138-4074-b22d-4b34e3b694c5","Type":"ContainerStarted","Data":"5017de9eec922d93474080f6c061f154e2921b046d7c141e85fd4f616c2b5243"}
Sep 08 18:33:09.407116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:09.406779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:09.422776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:09.422707    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4" podStartSLOduration=1.671694371 podStartE2EDuration="6.422692184s" totalImagesPullingTime="4.750997813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:33:09.422172172 +0000 UTC m=+563.873747303" watchObservedRunningTime="2026-09-08 18:33:09.422692184 +0000 UTC m=+563.874267315"
Sep 08 18:33:15.441332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:15.441301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2h7f4"
Sep 08 18:33:16.369750 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.368912    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:33:16.375583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.375553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.378179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.378151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:33:16.379558 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.379540    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lvhv4\""
Sep 08 18:33:16.379558 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.379540    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 18:33:16.503955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.503923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnk6k\" (UniqueName: \"kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.503955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.503959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.605232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.605192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnk6k\" (UniqueName: \"kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.605232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.605233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.607952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.607930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.613568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.613543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnk6k\" (UniqueName: \"kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k\") pod \"authorino-6f88f64db7-5lssv\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") " pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.686153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.686124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:33:16.959072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:16.959048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:33:16.961241 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:33:16.961218    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43394d0_09cc_4120_bb12_be6116a7d344.slice/crio-d57d087eef0d898e98d825b2f576fd7ed969b60911361f7322c6655068162f3d WatchSource:0}: Error finding container d57d087eef0d898e98d825b2f576fd7ed969b60911361f7322c6655068162f3d: Status 404 returned error can't find the container with id d57d087eef0d898e98d825b2f576fd7ed969b60911361f7322c6655068162f3d
Sep 08 18:33:17.442107 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:17.442078    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88f64db7-5lssv" event={"ID":"f43394d0-09cc-4120-bb12-be6116a7d344","Type":"ContainerStarted","Data":"d57d087eef0d898e98d825b2f576fd7ed969b60911361f7322c6655068162f3d"}
Sep 08 18:33:25.477416 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:25.477323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88f64db7-5lssv" event={"ID":"f43394d0-09cc-4120-bb12-be6116a7d344","Type":"ContainerStarted","Data":"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"}
Sep 08 18:33:25.495349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:33:25.495301    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f88f64db7-5lssv" podStartSLOduration=1.306776779 podStartE2EDuration="9.495285576s" totalImagesPullingTime="8.188508797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:33:25.494514837 +0000 UTC m=+579.946089969" watchObservedRunningTime="2026-09-08 18:33:25.495285576 +0000 UTC m=+579.946860752"
Sep 08 18:34:06.938357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.938274    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"]
Sep 08 18:34:06.956489 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.956443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"]
Sep 08 18:34:06.956674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.956547    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:06.959837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.959807    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4rjh4\""
Sep 08 18:34:06.959999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.959976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 18:34:06.969910 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.969887    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"]
Sep 08 18:34:06.972216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.972195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:06.985122 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:06.985094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"]
Sep 08 18:34:07.024089 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.024053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c80b8a-a736-4f60-88f8-c80e53c864ec-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.024280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.024117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575a15a2-860b-40d6-b80e-1e9849564dab-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.024280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.024191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8sf\" (UniqueName: \"kubernetes.io/projected/575a15a2-860b-40d6-b80e-1e9849564dab-kube-api-access-hx8sf\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.024392 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.024331    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gphfd\" (UniqueName: \"kubernetes.io/projected/31c80b8a-a736-4f60-88f8-c80e53c864ec-kube-api-access-gphfd\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.124759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.124702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c80b8a-a736-4f60-88f8-c80e53c864ec-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.124944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.124772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575a15a2-860b-40d6-b80e-1e9849564dab-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.124944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.124812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8sf\" (UniqueName: \"kubernetes.io/projected/575a15a2-860b-40d6-b80e-1e9849564dab-kube-api-access-hx8sf\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.124944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.124892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gphfd\" (UniqueName: \"kubernetes.io/projected/31c80b8a-a736-4f60-88f8-c80e53c864ec-kube-api-access-gphfd\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.125804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.125780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c80b8a-a736-4f60-88f8-c80e53c864ec-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.125910 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.125776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/575a15a2-860b-40d6-b80e-1e9849564dab-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.133496 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.133466    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gphfd\" (UniqueName: \"kubernetes.io/projected/31c80b8a-a736-4f60-88f8-c80e53c864ec-kube-api-access-gphfd\") pod \"payload-pre-processing-59fbc47bf8-hrsx4\" (UID: \"31c80b8a-a736-4f60-88f8-c80e53c864ec\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.133626 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.133583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8sf\" (UniqueName: \"kubernetes.io/projected/575a15a2-860b-40d6-b80e-1e9849564dab-kube-api-access-hx8sf\") pod \"payload-processing-5b54b7fbb7-2lwjs\" (UID: \"575a15a2-860b-40d6-b80e-1e9849564dab\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.269924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.269837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:07.281873 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.281846    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:07.547516 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.547450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"]
Sep 08 18:34:07.549775 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:34:07.549743    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a15a2_860b_40d6_b80e_1e9849564dab.slice/crio-415a9647d64fa5a1f5eb8321c7a99cf3c80365576f68403ac94c0c331e7bdb84 WatchSource:0}: Error finding container 415a9647d64fa5a1f5eb8321c7a99cf3c80365576f68403ac94c0c331e7bdb84: Status 404 returned error can't find the container with id 415a9647d64fa5a1f5eb8321c7a99cf3c80365576f68403ac94c0c331e7bdb84
Sep 08 18:34:07.562876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.562851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"]
Sep 08 18:34:07.566639 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:34:07.566611    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c80b8a_a736_4f60_88f8_c80e53c864ec.slice/crio-d9cf30c7476bd19779fec81b780ff7e6d1817d1d3c9b40b5f13c4ce22524b79e WatchSource:0}: Error finding container d9cf30c7476bd19779fec81b780ff7e6d1817d1d3c9b40b5f13c4ce22524b79e: Status 404 returned error can't find the container with id d9cf30c7476bd19779fec81b780ff7e6d1817d1d3c9b40b5f13c4ce22524b79e
Sep 08 18:34:07.660951 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.660910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4" event={"ID":"31c80b8a-a736-4f60-88f8-c80e53c864ec","Type":"ContainerStarted","Data":"d9cf30c7476bd19779fec81b780ff7e6d1817d1d3c9b40b5f13c4ce22524b79e"}
Sep 08 18:34:07.663001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:07.662970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs" event={"ID":"575a15a2-860b-40d6-b80e-1e9849564dab","Type":"ContainerStarted","Data":"415a9647d64fa5a1f5eb8321c7a99cf3c80365576f68403ac94c0c331e7bdb84"}
Sep 08 18:34:10.697972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.697933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs" event={"ID":"575a15a2-860b-40d6-b80e-1e9849564dab","Type":"ContainerStarted","Data":"70f2d47985e2506894b777dcedcc4128ea111c5a5bbfdb0e2e016a36afd1006c"}
Sep 08 18:34:10.698555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.698536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:10.699886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.699861    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4" event={"ID":"31c80b8a-a736-4f60-88f8-c80e53c864ec","Type":"ContainerStarted","Data":"031600a9ce3975a32ec43bf3417203e78f30d557a96bed280c76b04f5345f0a2"}
Sep 08 18:34:10.700261 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.700237    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:10.717112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.717066    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs" podStartSLOduration=2.3182958510000002 podStartE2EDuration="4.717051857s" totalImagesPullingTime="2.398756006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:10.714578558 +0000 UTC m=+625.166153691" watchObservedRunningTime="2026-09-08 18:34:10.717051857 +0000 UTC m=+625.168626986"
Sep 08 18:34:10.731501 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:10.731452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4" podStartSLOduration=2.3558456 podStartE2EDuration="4.731440445s" totalImagesPullingTime="2.375594845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:10.729290558 +0000 UTC m=+625.180865728" watchObservedRunningTime="2026-09-08 18:34:10.731440445 +0000 UTC m=+625.183015625"
Sep 08 18:34:16.727618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.727586    2582 generic.go:356] "Generic (PLEG): container finished" podID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerID="71d6818eaed2f65fbeff805765b71c8434e92790a7f59f9b162e739184c95720" exitCode=0
Sep 08 18:34:16.728081 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.727672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" event={"ID":"245faeeb-9478-415a-b635-cfbee2b8cdd2","Type":"ContainerDied","Data":"71d6818eaed2f65fbeff805765b71c8434e92790a7f59f9b162e739184c95720"}
Sep 08 18:34:16.728126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.728082    2582 scope.go:117] "RemoveContainer" containerID="71d6818eaed2f65fbeff805765b71c8434e92790a7f59f9b162e739184c95720"
Sep 08 18:34:16.729046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.729023    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:34:16.729103 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.729088    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:34:16.729139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:16.729118    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:34:17.734391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:17.734356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" event={"ID":"245faeeb-9478-415a-b635-cfbee2b8cdd2","Type":"ContainerStarted","Data":"990383e5c4c8484ae62ca2e230aca2a61ca9a1154113f89d3e92a4277d2be088"}
Sep 08 18:34:18.043849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:18.043752    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:34:18.044183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:18.044160    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 08 18:34:18.044243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:18.044211    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podUID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 08 18:34:19.043957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:19.043923    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 08 18:34:19.044342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:19.043984    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podUID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 08 18:34:20.043920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:20.043880    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 08 18:34:20.044091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:20.043939    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podUID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 08 18:34:21.043951 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:21.043913    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 08 18:34:21.044125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:21.043983    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podUID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 08 18:34:22.043165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:22.043121    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tmgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body=
Sep 08 18:34:22.043336 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:22.043185    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr" podUID="245faeeb-9478-415a-b635-cfbee2b8cdd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused"
Sep 08 18:34:22.711503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:22.711466    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2lwjs"
Sep 08 18:34:22.711995 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:22.711541    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-hrsx4"
Sep 08 18:34:23.047210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:23.047129    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:34:23.047409 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:23.047386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:34:23.047973 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:34:23.047954    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tmgsr"
Sep 08 18:35:15.816228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.816191    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"]
Sep 08 18:35:15.819791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.819766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.824494 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.824460    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 18:35:15.824494 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.824477    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 18:35:15.824710 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.824467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2gzvd\""
Sep 08 18:35:15.824710 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.824528    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 18:35:15.831375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.831355    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"]
Sep 08 18:35:15.909837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.909837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.910029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.910029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmd7\" (UniqueName: \"kubernetes.io/projected/6242c40e-0f64-4a81-a7ae-a75d62db53e3-kube-api-access-mwmd7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.910029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:15.910029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:15.909978    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011134 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011134 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmd7\" (UniqueName: \"kubernetes.io/projected/6242c40e-0f64-4a81-a7ae-a75d62db53e3-kube-api-access-mwmd7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011672    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.011891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.011699    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.013718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.013696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6242c40e-0f64-4a81-a7ae-a75d62db53e3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.013915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.013900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6242c40e-0f64-4a81-a7ae-a75d62db53e3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.020453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.020432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmd7\" (UniqueName: \"kubernetes.io/projected/6242c40e-0f64-4a81-a7ae-a75d62db53e3-kube-api-access-mwmd7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-m6dfl\" (UID: \"6242c40e-0f64-4a81-a7ae-a75d62db53e3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.131038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.130958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:16.404504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.404477    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"]
Sep 08 18:35:16.406682 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:35:16.406643    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6242c40e_0f64_4a81_a7ae_a75d62db53e3.slice/crio-8b8622d6c442ab7ddc85145e0ae2cee24ae5aebc7fa0776483c956eaa60a9a36 WatchSource:0}: Error finding container 8b8622d6c442ab7ddc85145e0ae2cee24ae5aebc7fa0776483c956eaa60a9a36: Status 404 returned error can't find the container with id 8b8622d6c442ab7ddc85145e0ae2cee24ae5aebc7fa0776483c956eaa60a9a36
Sep 08 18:35:16.408014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.407994    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:35:16.998978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:16.998898    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl" event={"ID":"6242c40e-0f64-4a81-a7ae-a75d62db53e3","Type":"ContainerStarted","Data":"8b8622d6c442ab7ddc85145e0ae2cee24ae5aebc7fa0776483c956eaa60a9a36"}
Sep 08 18:35:18.005236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:18.005194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl" event={"ID":"6242c40e-0f64-4a81-a7ae-a75d62db53e3","Type":"ContainerStarted","Data":"7f35112f1e8918d589b7ef9bc91444707511d6e1c2720875683122e415d34432"}
Sep 08 18:35:18.026382 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:18.026332    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl" podStartSLOduration=1.794833141 podStartE2EDuration="3.026315745s" totalImagesPullingTime="1.231482604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:18.023903983 +0000 UTC m=+692.475479115" watchObservedRunningTime="2026-09-08 18:35:18.026315745 +0000 UTC m=+692.477890878"
Sep 08 18:35:26.131257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:26.131217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:26.131777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:26.131341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:26.143701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:26.143679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:27.056073 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:27.056045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-m6dfl"
Sep 08 18:35:27.924932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:27.924899    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"]
Sep 08 18:35:27.927280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:27.927257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:27.930351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:27.930327    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 18:35:27.941135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:27.941104    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"]
Sep 08 18:35:28.008036 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.007998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.008036 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.008033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.008214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.008054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk45f\" (UniqueName: \"kubernetes.io/projected/1753665c-0bd5-4c47-83d3-bec38f004adb-kube-api-access-sk45f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.008214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.008124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753665c-0bd5-4c47-83d3-bec38f004adb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.008214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.008186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.008214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.008205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk45f\" (UniqueName: \"kubernetes.io/projected/1753665c-0bd5-4c47-83d3-bec38f004adb-kube-api-access-sk45f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753665c-0bd5-4c47-83d3-bec38f004adb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109651 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.109928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.109847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.111702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.111681    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1753665c-0bd5-4c47-83d3-bec38f004adb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.111871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.111855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1753665c-0bd5-4c47-83d3-bec38f004adb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.120435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.120404    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk45f\" (UniqueName: \"kubernetes.io/projected/1753665c-0bd5-4c47-83d3-bec38f004adb-kube-api-access-sk45f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lbjbk\" (UID: \"1753665c-0bd5-4c47-83d3-bec38f004adb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.238123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.238031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:28.506779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:28.506752    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"]
Sep 08 18:35:28.509098 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:35:28.509072    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1753665c_0bd5_4c47_83d3_bec38f004adb.slice/crio-9ced3d5f527c13c1eeab7b926e22244d3c8cc244fffae84a6c4b444e69364916 WatchSource:0}: Error finding container 9ced3d5f527c13c1eeab7b926e22244d3c8cc244fffae84a6c4b444e69364916: Status 404 returned error can't find the container with id 9ced3d5f527c13c1eeab7b926e22244d3c8cc244fffae84a6c4b444e69364916
Sep 08 18:35:29.062305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:29.062265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk" event={"ID":"1753665c-0bd5-4c47-83d3-bec38f004adb","Type":"ContainerStarted","Data":"f9d147504b4b47c6ddceb5ef7686fd7c62dc386a78a792cb8e039f1f70fa5252"}
Sep 08 18:35:29.062691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:29.062354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk" event={"ID":"1753665c-0bd5-4c47-83d3-bec38f004adb","Type":"ContainerStarted","Data":"9ced3d5f527c13c1eeab7b926e22244d3c8cc244fffae84a6c4b444e69364916"}
Sep 08 18:35:29.092012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:29.091964    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk" podStartSLOduration=1.901072587 podStartE2EDuration="2.091949521s" totalImagesPullingTime="190.876934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:29.091352071 +0000 UTC m=+703.542927202" watchObservedRunningTime="2026-09-08 18:35:29.091949521 +0000 UTC m=+703.543524649"
Sep 08 18:35:38.238185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:38.238095    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:38.238185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:38.238170    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:38.250415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:38.250388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:35:39.121982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:35:39.121943    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lbjbk"
Sep 08 18:36:31.273430 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.273396    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:36:31.273968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.273628    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f88f64db7-5lssv" podUID="f43394d0-09cc-4120-bb12-be6116a7d344" containerName="authorino" containerID="cri-o://c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749" gracePeriod=30
Sep 08 18:36:31.526980 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.526923    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:36:31.546958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.546930    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert\") pod \"f43394d0-09cc-4120-bb12-be6116a7d344\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") "
Sep 08 18:36:31.547095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.546972    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnk6k\" (UniqueName: \"kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k\") pod \"f43394d0-09cc-4120-bb12-be6116a7d344\" (UID: \"f43394d0-09cc-4120-bb12-be6116a7d344\") "
Sep 08 18:36:31.549139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.549106    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k" (OuterVolumeSpecName: "kube-api-access-hnk6k") pod "f43394d0-09cc-4120-bb12-be6116a7d344" (UID: "f43394d0-09cc-4120-bb12-be6116a7d344"). InnerVolumeSpecName "kube-api-access-hnk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:36:31.557068 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.557036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f43394d0-09cc-4120-bb12-be6116a7d344" (UID: "f43394d0-09cc-4120-bb12-be6116a7d344"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:36:31.648060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.648025    2582 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f43394d0-09cc-4120-bb12-be6116a7d344-tls-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:36:31.648060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:31.648050    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnk6k\" (UniqueName: \"kubernetes.io/projected/f43394d0-09cc-4120-bb12-be6116a7d344-kube-api-access-hnk6k\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:36:32.349415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.349374    2582 generic.go:356] "Generic (PLEG): container finished" podID="f43394d0-09cc-4120-bb12-be6116a7d344" containerID="c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749" exitCode=0
Sep 08 18:36:32.349855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.349422    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88f64db7-5lssv"
Sep 08 18:36:32.349855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.349422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88f64db7-5lssv" event={"ID":"f43394d0-09cc-4120-bb12-be6116a7d344","Type":"ContainerDied","Data":"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"}
Sep 08 18:36:32.349855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.349469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88f64db7-5lssv" event={"ID":"f43394d0-09cc-4120-bb12-be6116a7d344","Type":"ContainerDied","Data":"d57d087eef0d898e98d825b2f576fd7ed969b60911361f7322c6655068162f3d"}
Sep 08 18:36:32.349855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.349490    2582 scope.go:117] "RemoveContainer" containerID="c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"
Sep 08 18:36:32.359752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.359708    2582 scope.go:117] "RemoveContainer" containerID="c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"
Sep 08 18:36:32.360012 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:36:32.359990    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749\": container with ID starting with c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749 not found: ID does not exist" containerID="c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"
Sep 08 18:36:32.360102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.360018    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749"} err="failed to get container status \"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749\": rpc error: code = NotFound desc = could not find container \"c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749\": container with ID starting with c3666b702a37bafa43d351244373275b613279f58747566450de0922b89ba749 not found: ID does not exist"
Sep 08 18:36:32.381975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.381945    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:36:32.387187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:32.387165    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f88f64db7-5lssv"]
Sep 08 18:36:34.104184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:34.104150    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43394d0-09cc-4120-bb12-be6116a7d344" path="/var/lib/kubelet/pods/f43394d0-09cc-4120-bb12-be6116a7d344/volumes"
Sep 08 18:36:48.796037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.795992    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:36:48.796590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.796572    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43394d0-09cc-4120-bb12-be6116a7d344" containerName="authorino"
Sep 08 18:36:48.796699 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.796592    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43394d0-09cc-4120-bb12-be6116a7d344" containerName="authorino"
Sep 08 18:36:48.796777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.796699    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43394d0-09cc-4120-bb12-be6116a7d344" containerName="authorino"
Sep 08 18:36:48.801439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.801417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.804401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.804379    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1d8cb1-openshift-default-dockercfg-q5m5j\""
Sep 08 18:36:48.813276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.813252    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:36:48.870879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.870828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.870879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.870880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.870899    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.870960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.870977    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s82n\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.871050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.871102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.871135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.871216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.871181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972314 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s82n\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.972822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972811    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.973021 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.972893    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.973082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.973046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.973082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.973067    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.973187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.973122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.975172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.975138    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.975406 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.975384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.981163 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.981139    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:48.981254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:48.981214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s82n\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n\") pod \"e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") " pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:49.114717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.114637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:49.390761 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.390706    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:36:49.392112 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:49.392085    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dba3e7_e068_491a_beb1_315f1e29a7b5.slice/crio-a62edc928baf7275f0244a7387d932bf1c1fd39d668660af30f748d1f9aa6769 WatchSource:0}: Error finding container a62edc928baf7275f0244a7387d932bf1c1fd39d668660af30f748d1f9aa6769: Status 404 returned error can't find the container with id a62edc928baf7275f0244a7387d932bf1c1fd39d668660af30f748d1f9aa6769
Sep 08 18:36:49.394141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.394112    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:49.394217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.394177    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:49.394217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.394205    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:49.432321 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:49.432285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerStarted","Data":"a62edc928baf7275f0244a7387d932bf1c1fd39d668660af30f748d1f9aa6769"}
Sep 08 18:36:50.438635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.438596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerStarted","Data":"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"}
Sep 08 18:36:50.461159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.461102    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podStartSLOduration=2.461086234 podStartE2EDuration="2.461086234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:50.459457871 +0000 UTC m=+784.911033003" watchObservedRunningTime="2026-09-08 18:36:50.461086234 +0000 UTC m=+784.912661366"
Sep 08 18:36:50.504770 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.504733    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7587-76c4b5"]
Sep 08 18:36:50.508273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.508257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:50.511359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.511338    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhbkz\""
Sep 08 18:36:50.523934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.519534    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-76c4b5"]
Sep 08 18:36:50.585322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.585283    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx22l\" (UniqueName: \"kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l\") pod \"test-curl-7587-76c4b5\" (UID: \"87ef533b-74b3-4274-9663-d4f48fcf4b6f\") " pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:50.686627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.686585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx22l\" (UniqueName: \"kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l\") pod \"test-curl-7587-76c4b5\" (UID: \"87ef533b-74b3-4274-9663-d4f48fcf4b6f\") " pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:50.698421 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.698345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx22l\" (UniqueName: \"kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l\") pod \"test-curl-7587-76c4b5\" (UID: \"87ef533b-74b3-4274-9663-d4f48fcf4b6f\") " pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:50.818944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:50.818901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:51.091953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.091928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7587-76c4b5"]
Sep 08 18:36:51.093794 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:51.093767    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87ef533b_74b3_4274_9663_d4f48fcf4b6f.slice/crio-8c98f681edffd143ac2eb9812805e8d602196b63a2866cf180f60dab1c038f1e WatchSource:0}: Error finding container 8c98f681edffd143ac2eb9812805e8d602196b63a2866cf180f60dab1c038f1e: Status 404 returned error can't find the container with id 8c98f681edffd143ac2eb9812805e8d602196b63a2866cf180f60dab1c038f1e
Sep 08 18:36:51.115410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.115384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:51.120391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.120365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:51.444071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.444036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-76c4b5" event={"ID":"87ef533b-74b3-4274-9663-d4f48fcf4b6f","Type":"ContainerStarted","Data":"8c98f681edffd143ac2eb9812805e8d602196b63a2866cf180f60dab1c038f1e"}
Sep 08 18:36:51.444576 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.444371    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:51.445525 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:51.445505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:36:52.449745 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:52.449676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7587-76c4b5" event={"ID":"87ef533b-74b3-4274-9663-d4f48fcf4b6f","Type":"ContainerStarted","Data":"2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234"}
Sep 08 18:36:52.466165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:52.466116    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7587-76c4b5" podStartSLOduration=1.597801778 podStartE2EDuration="2.466095941s" totalImagesPullingTime="868.294163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:52.465607955 +0000 UTC m=+786.917183089" watchObservedRunningTime="2026-09-08 18:36:52.466095941 +0000 UTC m=+786.917671124"
Sep 08 18:36:53.069330 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:53.069288    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7587-76c4b5"]
Sep 08 18:36:53.072659 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:53.072628    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7587-76c4b5"]
Sep 08 18:36:53.453992 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:36:53.453959    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7587-76c4b5" container="test-curl-7587-76c4b5"
Sep 08 18:36:54.231133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.231098    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:36:54.231460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.231448    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" containerName="test-curl-7587-76c4b5"
Sep 08 18:36:54.231508 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.231461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" containerName="test-curl-7587-76c4b5"
Sep 08 18:36:54.231542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.231535    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" containerName="test-curl-7587-76c4b5"
Sep 08 18:36:54.234789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.234769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.237398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.237378    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1d8cb1\""
Sep 08 18:36:54.246117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.246093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:36:54.266014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.265983    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:36:54.269361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.269344    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.272899 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.272880    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1d8cb1-dockercfg-r7jft\""
Sep 08 18:36:54.282622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.282599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:36:54.318939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.318900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgv5\" (UniqueName: \"kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.318939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.318941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.319157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.319012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtx8\" (UniqueName: \"kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.319157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.319034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.419518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.419483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtx8\" (UniqueName: \"kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.419718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.419530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.419718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.419569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgv5\" (UniqueName: \"kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.419718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.419607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.420550 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.420435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.420830 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.420814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.429141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.429115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgv5\" (UniqueName: \"kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5\") pod \"payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.429289 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.429267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtx8\" (UniqueName: \"kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8\") pod \"payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.458132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.458095    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7587-76c4b5" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" containerName="test-curl-7587-76c4b5" containerID="cri-o://2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234" gracePeriod=2
Sep 08 18:36:54.546492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.546410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:54.579062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.579033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:54.824936 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.824909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:36:54.826692 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:54.826670    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d28bfa_b61d_43b1_88c2_6f34de110d8c.slice/crio-22f91c9d541633b5cb5d57f49a7a52d5b929c7c346cfa3d5d846253236777557 WatchSource:0}: Error finding container 22f91c9d541633b5cb5d57f49a7a52d5b929c7c346cfa3d5d846253236777557: Status 404 returned error can't find the container with id 22f91c9d541633b5cb5d57f49a7a52d5b929c7c346cfa3d5d846253236777557
Sep 08 18:36:54.862100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:54.862075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:36:54.865278 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:54.865248    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729c07d3_cb25_4322_ac62_8eccfc3bdc51.slice/crio-e2dec2b3be1cc43dc0489f1afeb7e077a2677d10ca9555a44832b16a08a6c7e9 WatchSource:0}: Error finding container e2dec2b3be1cc43dc0489f1afeb7e077a2677d10ca9555a44832b16a08a6c7e9: Status 404 returned error can't find the container with id e2dec2b3be1cc43dc0489f1afeb7e077a2677d10ca9555a44832b16a08a6c7e9
Sep 08 18:36:55.465620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.465588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" event={"ID":"f7d28bfa-b61d-43b1-88c2-6f34de110d8c","Type":"ContainerStarted","Data":"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"}
Sep 08 18:36:55.466097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.465642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" event={"ID":"f7d28bfa-b61d-43b1-88c2-6f34de110d8c","Type":"ContainerStarted","Data":"22f91c9d541633b5cb5d57f49a7a52d5b929c7c346cfa3d5d846253236777557"}
Sep 08 18:36:55.466097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.465705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:36:55.467354 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.467323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" event={"ID":"729c07d3-cb25-4322-ac62-8eccfc3bdc51","Type":"ContainerStarted","Data":"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"}
Sep 08 18:36:55.467453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.467389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" event={"ID":"729c07d3-cb25-4322-ac62-8eccfc3bdc51","Type":"ContainerStarted","Data":"e2dec2b3be1cc43dc0489f1afeb7e077a2677d10ca9555a44832b16a08a6c7e9"}
Sep 08 18:36:55.467453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.467416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:36:55.483652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.483604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" podStartSLOduration=1.483587848 podStartE2EDuration="1.483587848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:55.48215317 +0000 UTC m=+789.933728299" watchObservedRunningTime="2026-09-08 18:36:55.483587848 +0000 UTC m=+789.935162979"
Sep 08 18:36:55.500369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:55.500311    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" podStartSLOduration=1.500292649 podStartE2EDuration="1.500292649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:55.499092118 +0000 UTC m=+789.950667250" watchObservedRunningTime="2026-09-08 18:36:55.500292649 +0000 UTC m=+789.951867809"
Sep 08 18:36:56.649665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:56.649640    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:56.739528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:56.739440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx22l\" (UniqueName: \"kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l\") pod \"87ef533b-74b3-4274-9663-d4f48fcf4b6f\" (UID: \"87ef533b-74b3-4274-9663-d4f48fcf4b6f\") "
Sep 08 18:36:56.741808 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:56.741774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l" (OuterVolumeSpecName: "kube-api-access-bx22l") pod "87ef533b-74b3-4274-9663-d4f48fcf4b6f" (UID: "87ef533b-74b3-4274-9663-d4f48fcf4b6f"). InnerVolumeSpecName "kube-api-access-bx22l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:36:56.840946 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:56.840912    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx22l\" (UniqueName: \"kubernetes.io/projected/87ef533b-74b3-4274-9663-d4f48fcf4b6f-kube-api-access-bx22l\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:36:57.013513 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.013422    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:36:57.017328 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.017307    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.021221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.021196    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-abd9a747-openshift-default-dockercfg-v9s99\""
Sep 08 18:36:57.034238 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.034214    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:36:57.042924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.042894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597m7\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.042943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.042981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.043269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.043212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144380 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144796 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144796 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597m7\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144796 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.144944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.144834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.145077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.145047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.145172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.145147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.145250 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.145206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.145250 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.145217    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.145509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.145294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.147196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.147165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.147275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.147208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.162934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.162906    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.163157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.163135    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-597m7\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7\") pod \"e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") " pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.330714 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.330628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:57.479791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.479759    2582 generic.go:356] "Generic (PLEG): container finished" podID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" containerID="2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234" exitCode=137
Sep 08 18:36:57.479944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.479844    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:57.479944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.479876    2582 scope.go:117] "RemoveContainer" containerID="2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234"
Sep 08 18:36:57.489240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.489220    2582 scope.go:117] "RemoveContainer" containerID="2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234"
Sep 08 18:36:57.489535 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:36:57.489513    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234\": container with ID starting with 2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234 not found: ID does not exist" containerID="2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234"
Sep 08 18:36:57.489589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.489548    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234"} err="failed to get container status \"2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234\": rpc error: code = NotFound desc = could not find container \"2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234\": container with ID starting with 2afd981e03e0268ea337fb39ba1484fe25ce4c85982b2a4e4009a2d539618234 not found: ID does not exist"
Sep 08 18:36:57.490864 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:36:57.490844    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7587-76c4b5\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:57.609183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.609157    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:36:57.610874 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:57.610852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef14cb66_33b7_4619_8eaa_d6706113fb3b.slice/crio-116f4e423487fa1ac7e29b0fe287fca4ad5c6125dc2a719aa1e208645c6802a0 WatchSource:0}: Error finding container 116f4e423487fa1ac7e29b0fe287fca4ad5c6125dc2a719aa1e208645c6802a0: Status 404 returned error can't find the container with id 116f4e423487fa1ac7e29b0fe287fca4ad5c6125dc2a719aa1e208645c6802a0
Sep 08 18:36:57.613259 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.613222    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:57.613350 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.613333    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:57.613394 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.613380    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:57.792780 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.792740    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:36:57.796887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.796861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.802452 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.801600    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0eaf99-openshift-default-dockercfg-jlv4l\""
Sep 08 18:36:57.813196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.812816    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:36:57.831483 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:36:57.830811    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7587-76c4b5\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" pod="opendatahub/test-curl-7587-76c4b5"
Sep 08 18:36:57.852283 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852447 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852322    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852447 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852447 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zjl\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.852831 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.852639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.953977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zjl\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954367 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954367 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954367 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954603 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.954945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.955046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.955046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.955046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.954995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.956888 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.956865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.957272 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.957253    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.963957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.963930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:57.964081 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:57.964064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zjl\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl\") pod \"e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") " pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:58.100909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.100868    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87ef533b-74b3-4274-9663-d4f48fcf4b6f" path="/var/lib/kubelet/pods/87ef533b-74b3-4274-9663-d4f48fcf4b6f/volumes"
Sep 08 18:36:58.116663 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.116634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:36:58.405629 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.405535    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:36:58.409434 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:36:58.409402    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7687c06_8b91_41d0_95f8_42fdb9be3d53.slice/crio-066de7701bc083664af07474346261656f2d6a8838779e057b9a3cbf6454ae23 WatchSource:0}: Error finding container 066de7701bc083664af07474346261656f2d6a8838779e057b9a3cbf6454ae23: Status 404 returned error can't find the container with id 066de7701bc083664af07474346261656f2d6a8838779e057b9a3cbf6454ae23
Sep 08 18:36:58.411879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.411847    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:58.413763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.412323    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:58.413763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.412462    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:36:58.495748 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.494109    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerStarted","Data":"066de7701bc083664af07474346261656f2d6a8838779e057b9a3cbf6454ae23"}
Sep 08 18:36:58.499548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.496031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerStarted","Data":"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"}
Sep 08 18:36:58.499548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.496085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerStarted","Data":"116f4e423487fa1ac7e29b0fe287fca4ad5c6125dc2a719aa1e208645c6802a0"}
Sep 08 18:36:58.519788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:58.519717    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podStartSLOduration=2.519700878 podStartE2EDuration="2.519700878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:58.517861262 +0000 UTC m=+792.969436396" watchObservedRunningTime="2026-09-08 18:36:58.519700878 +0000 UTC m=+792.971276010"
Sep 08 18:36:59.330920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.330877    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:59.336080 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.336053    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:59.504500 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.504460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerStarted","Data":"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"}
Sep 08 18:36:59.504756 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.504686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:59.505771 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.505714    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:36:59.526054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:36:59.526006    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podStartSLOduration=2.525990861 podStartE2EDuration="2.525990861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:59.52399563 +0000 UTC m=+793.975570760" watchObservedRunningTime="2026-09-08 18:36:59.525990861 +0000 UTC m=+793.977565993"
Sep 08 18:37:00.117466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:00.117430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:00.123027 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:00.122998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:00.509359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:00.509326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:00.510429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:00.510407    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:01.808877 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.808845    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:01.813566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.813545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:01.816737 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.816703    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-abd9a747-dockercfg-6cjwj\""
Sep 08 18:37:01.816901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.816881    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-abd9a747\""
Sep 08 18:37:01.824545 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.824521    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:01.903481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.903446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:01.903665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:01.903578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgdz\" (UniqueName: \"kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.004503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.004462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgdz\" (UniqueName: \"kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.004790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.004514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.005198 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.005179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.017827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.017801    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgdz\" (UniqueName: \"kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz\") pod \"payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.125276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.125187    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.423462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.423436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:02.425541 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:37:02.425512    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa880a59_334e_4eb7_a961_203632be133f.slice/crio-8bbdbf6e8d6813286213dac8ad1326dcd304188224fc702b2422da316507eece WatchSource:0}: Error finding container 8bbdbf6e8d6813286213dac8ad1326dcd304188224fc702b2422da316507eece: Status 404 returned error can't find the container with id 8bbdbf6e8d6813286213dac8ad1326dcd304188224fc702b2422da316507eece
Sep 08 18:37:02.521698 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.521660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" event={"ID":"fa880a59-334e-4eb7-a961-203632be133f","Type":"ContainerStarted","Data":"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"}
Sep 08 18:37:02.521847 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.521739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" event={"ID":"fa880a59-334e-4eb7-a961-203632be133f","Type":"ContainerStarted","Data":"8bbdbf6e8d6813286213dac8ad1326dcd304188224fc702b2422da316507eece"}
Sep 08 18:37:02.522243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.522217    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:02.540635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:02.540577    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" podStartSLOduration=1.540556196 podStartE2EDuration="1.540556196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:02.538992006 +0000 UTC m=+796.990567138" watchObservedRunningTime="2026-09-08 18:37:02.540556196 +0000 UTC m=+796.992131330"
Sep 08 18:37:03.331078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.331037    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:37:03.334897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.334876    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.338397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.338374    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-0eaf99-dockercfg-vc82r\""
Sep 08 18:37:03.338637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.338622    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0eaf99\""
Sep 08 18:37:03.346154 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.346130    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:37:03.417616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.417581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9bk\" (UniqueName: \"kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.417802 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.417628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.518531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.518488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9bk\" (UniqueName: \"kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.518752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.518541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.519291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.519272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.526879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.526850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9bk\" (UniqueName: \"kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk\") pod \"payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.648002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.647967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:03.936984 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:03.936961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:37:03.939459 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:37:03.939432    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166a2a81_d758_44a3_88df_4a5ba56eb677.slice/crio-7d5847c9a68eadda249d0271069bdc055e4d93e48a5da48b92d359d931158f08 WatchSource:0}: Error finding container 7d5847c9a68eadda249d0271069bdc055e4d93e48a5da48b92d359d931158f08: Status 404 returned error can't find the container with id 7d5847c9a68eadda249d0271069bdc055e4d93e48a5da48b92d359d931158f08
Sep 08 18:37:04.533839 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:04.533799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" event={"ID":"166a2a81-d758-44a3-88df-4a5ba56eb677","Type":"ContainerStarted","Data":"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"}
Sep 08 18:37:04.534247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:04.533869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" event={"ID":"166a2a81-d758-44a3-88df-4a5ba56eb677","Type":"ContainerStarted","Data":"7d5847c9a68eadda249d0271069bdc055e4d93e48a5da48b92d359d931158f08"}
Sep 08 18:37:04.534247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:04.533901    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:04.550328 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:04.550277    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" podStartSLOduration=1.550260998 podStartE2EDuration="1.550260998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:04.549862117 +0000 UTC m=+799.001437274" watchObservedRunningTime="2026-09-08 18:37:04.550260998 +0000 UTC m=+799.001836134"
Sep 08 18:37:06.475939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:06.475906    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:37:06.476411 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:06.475996    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:37:07.394425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.394315    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:37972->10.134.0.37:15021: read: connection reset by peer" start-of-body=
Sep 08 18:37:07.394425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.394372    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:37972->10.134.0.37:15021: read: connection reset by peer"
Sep 08 18:37:07.567453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.567363    2582 generic.go:356] "Generic (PLEG): container finished" podID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerID="039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b" exitCode=0
Sep 08 18:37:07.567962 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.567534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerDied","Data":"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"}
Sep 08 18:37:07.568033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.568008    2582 scope.go:117] "RemoveContainer" containerID="039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"
Sep 08 18:37:07.574074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.574028    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:07.574210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.574127    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:07.574210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:07.574177    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:08.578464 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:08.578402    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerStarted","Data":"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"}
Sep 08 18:37:09.115053 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:09.115018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:37:09.115230 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:09.115200    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 18:37:09.115279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:09.115241    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 18:37:10.116082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:10.116046    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 18:37:10.116537 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:10.116510    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 18:37:11.116028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:11.115986    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 18:37:11.116202 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:11.116053    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 18:37:12.115953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.115917    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 18:37:12.116135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.115981    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 18:37:12.934560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.934524    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:37:12.938841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.938815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:12.950276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.950244    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:37:12.950998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:12.950966    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-b3341dd9-kserve-self-signed-certs\""
Sep 08 18:37:13.034250 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.034210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d52q\" (UniqueName: \"kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.034591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.034568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.034797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.034778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.034981 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.034963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.035160 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.035145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.035310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.035294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.134769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.126192    2582 patch_prober.go:28] interesting pod/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 08 18:37:13.134769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.126296    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 08 18:37:13.139535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.139498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d52q\" (UniqueName: \"kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.140466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.139906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.140772 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.140752    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.141408 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.141370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.141498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.141464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.141498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.141494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.141614 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.141383    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.141959 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.141937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.142278 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.142235    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.143326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.143304    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.144549 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.144527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.148092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.148073    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d52q\" (UniqueName: \"kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q\") pod \"e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") " pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.270909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.270820    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:13.529221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.529137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:13.636810 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:13.636772    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:37:13.639369 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:37:13.639334    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda061d5ab_0aec_4d05_ba32_1e6b7af7fed7.slice/crio-390bd3ba9cc0bcd8435bbacda14ef33fb2fab36feb0f4658a0fa5ab8a5bee0cc WatchSource:0}: Error finding container 390bd3ba9cc0bcd8435bbacda14ef33fb2fab36feb0f4658a0fa5ab8a5bee0cc: Status 404 returned error can't find the container with id 390bd3ba9cc0bcd8435bbacda14ef33fb2fab36feb0f4658a0fa5ab8a5bee0cc
Sep 08 18:37:14.203981 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.203935    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:37:14.205135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.205102    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:37:14.206023 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.205999    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:37:14.626132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.626045    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" event={"ID":"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7","Type":"ContainerStarted","Data":"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"}
Sep 08 18:37:14.626132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.626093    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" event={"ID":"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7","Type":"ContainerStarted","Data":"390bd3ba9cc0bcd8435bbacda14ef33fb2fab36feb0f4658a0fa5ab8a5bee0cc"}
Sep 08 18:37:14.643852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:14.643805    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" podStartSLOduration=2.643788885 podStartE2EDuration="2.643788885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:14.642955731 +0000 UTC m=+809.094530865" watchObservedRunningTime="2026-09-08 18:37:14.643788885 +0000 UTC m=+809.095364017"
Sep 08 18:37:15.540503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:15.540467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:37:15.547889 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:15.547861    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 18:37:15.548008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:15.547923    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 18:37:17.644010 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.643976    2582 generic.go:356] "Generic (PLEG): container finished" podID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerID="16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a" exitCode=0
Sep 08 18:37:17.644534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.644053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerDied","Data":"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"}
Sep 08 18:37:17.644599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.644569    2582 scope.go:117] "RemoveContainer" containerID="16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"
Sep 08 18:37:17.645675 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.645647    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:17.645788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.645735    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:17.645788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:17.645768    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:18.651594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:18.651559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerStarted","Data":"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"}
Sep 08 18:37:19.332979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:19.331395    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:19.332979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:19.331641    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 18:37:19.332979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:19.331691    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 18:37:20.331639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:20.331596    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 18:37:20.332136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:20.331662    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 18:37:21.331838 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:21.331800    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 18:37:21.332221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:21.331865    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 18:37:22.331033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.330996    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 18:37:22.331233 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.331047    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 18:37:22.683150 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.683115    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerID="bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6" exitCode=0
Sep 08 18:37:22.683616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.683193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerDied","Data":"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"}
Sep 08 18:37:22.683664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.683639    2582 scope.go:117] "RemoveContainer" containerID="bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"
Sep 08 18:37:22.684925 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.684880    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:22.685023 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.685006    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:22.685066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:22.685047    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:23.272464 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.272416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:23.272664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.272482    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:23.292436 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.292402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:23.331179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.331141    2582 patch_prober.go:28] interesting pod/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 08 18:37:23.331352 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.331199    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 08 18:37:23.695334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.695297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerStarted","Data":"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"}
Sep 08 18:37:23.713775 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:23.713715    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:37:24.116997 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.116915    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:24.117169 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.117128    2582 patch_prober.go:28] interesting pod/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 08 18:37:24.117239 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.117183    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 08 18:37:24.334885 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.334850    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:24.335254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.335232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:24.335904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:24.335885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:25.117422 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:25.117391    2582 patch_prober.go:28] interesting pod/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 08 18:37:25.117834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:25.117444    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 08 18:37:26.117646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:26.117610    2582 patch_prober.go:28] interesting pod/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 08 18:37:26.118098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:26.117664    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 08 18:37:27.117450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:27.117416    2582 patch_prober.go:28] interesting pod/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 08 18:37:27.117607 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:27.117470    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 08 18:37:28.117540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:28.117501    2582 patch_prober.go:28] interesting pod/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 08 18:37:28.117966 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:28.117574    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 08 18:37:29.345391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:29.345345    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:29.345916 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:29.345761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:29.346947 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:29.346921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:37:42.047506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.047471    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:42.048093 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.047768    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" podUID="fa880a59-334e-4eb7-a961-203632be133f" containerName="payload-pre-processing" containerID="cri-o://ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153" gracePeriod=30
Sep 08 18:37:42.301896 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.301830    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:42.336739 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.336692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grgdz\" (UniqueName: \"kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz\") pod \"fa880a59-334e-4eb7-a961-203632be133f\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") "
Sep 08 18:37:42.336911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.336811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume\") pod \"fa880a59-334e-4eb7-a961-203632be133f\" (UID: \"fa880a59-334e-4eb7-a961-203632be133f\") "
Sep 08 18:37:42.337190 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.337159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa880a59-334e-4eb7-a961-203632be133f" (UID: "fa880a59-334e-4eb7-a961-203632be133f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:37:42.339095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.339063    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz" (OuterVolumeSpecName: "kube-api-access-grgdz") pod "fa880a59-334e-4eb7-a961-203632be133f" (UID: "fa880a59-334e-4eb7-a961-203632be133f"). InnerVolumeSpecName "kube-api-access-grgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:37:42.346899 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.346875    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:37:42.347105 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.347082    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy" containerID="cri-o://5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0" gracePeriod=30
Sep 08 18:37:42.437897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.437852    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa880a59-334e-4eb7-a961-203632be133f-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.437897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.437896    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgdz\" (UniqueName: \"kubernetes.io/projected/fa880a59-334e-4eb7-a961-203632be133f-kube-api-access-grgdz\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:42.793511 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.793473    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa880a59-334e-4eb7-a961-203632be133f" containerID="ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153" exitCode=0
Sep 08 18:37:42.793704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.793515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"
Sep 08 18:37:42.793704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.793525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" event={"ID":"fa880a59-334e-4eb7-a961-203632be133f","Type":"ContainerDied","Data":"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"}
Sep 08 18:37:42.793704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.793576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2" event={"ID":"fa880a59-334e-4eb7-a961-203632be133f","Type":"ContainerDied","Data":"8bbdbf6e8d6813286213dac8ad1326dcd304188224fc702b2422da316507eece"}
Sep 08 18:37:42.793704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.793602    2582 scope.go:117] "RemoveContainer" containerID="ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"
Sep 08 18:37:42.806037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.805813    2582 scope.go:117] "RemoveContainer" containerID="ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"
Sep 08 18:37:42.806601 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:37:42.806569    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153\": container with ID starting with ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153 not found: ID does not exist" containerID="ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"
Sep 08 18:37:42.806694 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.806611    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153"} err="failed to get container status \"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153\": rpc error: code = NotFound desc = could not find container \"ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153\": container with ID starting with ac87dd9048b47819323ac7171144907678917a35949887e7ccb76f6c3e38b153 not found: ID does not exist"
Sep 08 18:37:42.824407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.824380    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:42.828781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:42.828758    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-abd9a747-68bbbdd4b6-fpzx2"]
Sep 08 18:37:43.296696 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.296658    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:37:43.297092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.297078    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa880a59-334e-4eb7-a961-203632be133f" containerName="payload-pre-processing"
Sep 08 18:37:43.297135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.297094    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa880a59-334e-4eb7-a961-203632be133f" containerName="payload-pre-processing"
Sep 08 18:37:43.297186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.297177    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa880a59-334e-4eb7-a961-203632be133f" containerName="payload-pre-processing"
Sep 08 18:37:43.299905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.299886    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.302766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.302709    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ad9a714c-gw-openshift-default-dockercfg-tcqzm\""
Sep 08 18:37:43.313247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.313221    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:37:43.346327 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqbw\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346438    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.346794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.346608    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.447658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.447870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.447870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.447870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.447999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqbw\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448444 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.448539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.448519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.450606 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.450578    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.450917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.450898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.458236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.458210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.459126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.459101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqbw\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw\") pod \"e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") " pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.612263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.611839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:43.889608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.889580    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:37:43.891431 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:37:43.891406    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68107a5_f972_4436_89ff_971e3fbd24ed.slice/crio-dbf7593b77ebb102c6395ea80d6a4f5985c819684aad729949c356acc8656601 WatchSource:0}: Error finding container dbf7593b77ebb102c6395ea80d6a4f5985c819684aad729949c356acc8656601: Status 404 returned error can't find the container with id dbf7593b77ebb102c6395ea80d6a4f5985c819684aad729949c356acc8656601
Sep 08 18:37:43.893633 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.893602    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:43.893711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.893664    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:43.893711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:43.893692    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:37:44.099066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:44.099037    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa880a59-334e-4eb7-a961-203632be133f" path="/var/lib/kubelet/pods/fa880a59-334e-4eb7-a961-203632be133f/volumes"
Sep 08 18:37:44.805795 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:44.805761    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerStarted","Data":"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"}
Sep 08 18:37:44.806178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:44.805818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerStarted","Data":"dbf7593b77ebb102c6395ea80d6a4f5985c819684aad729949c356acc8656601"}
Sep 08 18:37:44.828512 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:44.828460    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podStartSLOduration=1.828444077 podStartE2EDuration="1.828444077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:37:44.825818547 +0000 UTC m=+839.277393679" watchObservedRunningTime="2026-09-08 18:37:44.828444077 +0000 UTC m=+839.280019208"
Sep 08 18:37:45.613119 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:45.613081    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:45.618516 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:45.618490    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:45.814490 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:45.814461    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:45.815636 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:45.815579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:37:47.604821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.604793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:47.691982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.691949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-597m7\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692088    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692114    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692157    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692184    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy\") pod \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\" (UID: \"ef14cb66-33b7-4619-8eaa-d6706113fb3b\") "
Sep 08 18:37:47.692427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692375    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:37:47.692626 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:37:47.692626 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692522    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:37:47.692751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:37:47.692751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692660    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:37:47.692751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692688    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.692751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692709    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.692895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.692762    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.694654 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.694625    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:37:47.694777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.694658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7" (OuterVolumeSpecName: "kube-api-access-597m7") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "kube-api-access-597m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:37:47.694777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.694698    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:37:47.694777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.694718    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef14cb66-33b7-4619-8eaa-d6706113fb3b" (UID: "ef14cb66-33b7-4619-8eaa-d6706113fb3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:37:47.793834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793791    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.793834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793825    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597m7\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-kube-api-access-597m7\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.793834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793835    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.793834 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793845    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.794102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793854    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef14cb66-33b7-4619-8eaa-d6706113fb3b-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.794102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.793862    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef14cb66-33b7-4619-8eaa-d6706113fb3b-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:37:47.825979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.825935    2582 generic.go:356] "Generic (PLEG): container finished" podID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerID="5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0" exitCode=0
Sep 08 18:37:47.826138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.825986    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"
Sep 08 18:37:47.826138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.825979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerDied","Data":"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"}
Sep 08 18:37:47.826138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.826118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld" event={"ID":"ef14cb66-33b7-4619-8eaa-d6706113fb3b","Type":"ContainerDied","Data":"116f4e423487fa1ac7e29b0fe287fca4ad5c6125dc2a719aa1e208645c6802a0"}
Sep 08 18:37:47.826321 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.826149    2582 scope.go:117] "RemoveContainer" containerID="5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"
Sep 08 18:37:47.835788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.835770    2582 scope.go:117] "RemoveContainer" containerID="16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"
Sep 08 18:37:47.843763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.843740    2582 scope.go:117] "RemoveContainer" containerID="5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"
Sep 08 18:37:47.844061 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:37:47.844042    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0\": container with ID starting with 5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0 not found: ID does not exist" containerID="5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"
Sep 08 18:37:47.844114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.844071    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0"} err="failed to get container status \"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0\": rpc error: code = NotFound desc = could not find container \"5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0\": container with ID starting with 5d4717163643d73e10a752a2b73612fc0d69703694f21563c6485a17448f3de0 not found: ID does not exist"
Sep 08 18:37:47.844114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.844087    2582 scope.go:117] "RemoveContainer" containerID="16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"
Sep 08 18:37:47.844347 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:37:47.844329    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a\": container with ID starting with 16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a not found: ID does not exist" containerID="16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"
Sep 08 18:37:47.844384 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.844354    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a"} err="failed to get container status \"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a\": rpc error: code = NotFound desc = could not find container \"16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a\": container with ID starting with 16a59e820d70bc2ce5467a2f309d250858078c0f9338d6fe417a8e87d042431a not found: ID does not exist"
Sep 08 18:37:47.849177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.849154    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:37:47.854454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:47.854430    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-abd9a747-openshift-default-675d69c7c8-mr8ld"]
Sep 08 18:37:48.099293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:37:48.099217    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" path="/var/lib/kubelet/pods/ef14cb66-33b7-4619-8eaa-d6706113fb3b/volumes"
Sep 08 18:38:06.929684 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.929647    2582 generic.go:356] "Generic (PLEG): container finished" podID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerID="5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20" exitCode=0
Sep 08 18:38:06.930191 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.929718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerDied","Data":"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"}
Sep 08 18:38:06.930191 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.930115    2582 scope.go:117] "RemoveContainer" containerID="5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"
Sep 08 18:38:06.931205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.931180    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:06.931259 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.931246    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:06.931297 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:06.931277    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:07.936555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:07.936516    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerStarted","Data":"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"}
Sep 08 18:38:08.613685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:08.613646    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:08.613884 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:08.613853    2582 patch_prober.go:28] interesting pod/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 08 18:38:08.613925 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:08.613909    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 08 18:38:09.613559 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:09.613514    2582 patch_prober.go:28] interesting pod/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 08 18:38:09.613992 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:09.613576    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 08 18:38:10.613794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:10.613751    2582 patch_prober.go:28] interesting pod/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 08 18:38:10.614172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:10.613825    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 08 18:38:11.223588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.223554    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"]
Sep 08 18:38:11.223996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.223983    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.224043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.223998    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.224043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.224010    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.224043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.224015    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.224132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.224082    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.224132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.224090    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef14cb66-33b7-4619-8eaa-d6706113fb3b" containerName="istio-proxy"
Sep 08 18:38:11.225955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.225939    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:11.228904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.228883    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:38:11.234883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.234859    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"]
Sep 08 18:38:11.322822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.322782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl\" (UID: \"c0a1943b-6ef7-4645-8686-644057c96ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:11.424292 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.424252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl\" (UID: \"c0a1943b-6ef7-4645-8686-644057c96ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:11.424948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.424930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl\" (UID: \"c0a1943b-6ef7-4645-8686-644057c96ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:11.538047 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.537973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:11.614015 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.613660    2582 patch_prober.go:28] interesting pod/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 08 18:38:11.614015 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.613710    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 08 18:38:11.835582 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.835557    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"]
Sep 08 18:38:11.837863 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:11.837835    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a1943b_6ef7_4645_8686_644057c96ea8.slice/crio-5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68 WatchSource:0}: Error finding container 5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68: Status 404 returned error can't find the container with id 5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68
Sep 08 18:38:11.958051 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:11.958007    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl" event={"ID":"c0a1943b-6ef7-4645-8686-644057c96ea8","Type":"ContainerStarted","Data":"5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68"}
Sep 08 18:38:12.612994 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:12.612946    2582 patch_prober.go:28] interesting pod/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 08 18:38:12.613212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:12.613010    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 08 18:38:13.617944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:13.617906    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:13.618361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:13.618287    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:13.619284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:13.619254    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:14.976217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:14.976182    2582 generic.go:356] "Generic (PLEG): container finished" podID="c0a1943b-6ef7-4645-8686-644057c96ea8" containerID="0e896ec988eb531e59a9c6bcec73c71291828bdba8740b0b0765a4a4922f9905" exitCode=0
Sep 08 18:38:14.976674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:14.976269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl" event={"ID":"c0a1943b-6ef7-4645-8686-644057c96ea8","Type":"ContainerDied","Data":"0e896ec988eb531e59a9c6bcec73c71291828bdba8740b0b0765a4a4922f9905"}
Sep 08 18:38:16.116243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.116217    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:16.264588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.264498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca\") pod \"c0a1943b-6ef7-4645-8686-644057c96ea8\" (UID: \"c0a1943b-6ef7-4645-8686-644057c96ea8\") "
Sep 08 18:38:16.264936 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.264914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0a1943b-6ef7-4645-8686-644057c96ea8" (UID: "c0a1943b-6ef7-4645-8686-644057c96ea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:16.365929 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.365890    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1943b-6ef7-4645-8686-644057c96ea8-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:16.987299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.987269    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"
Sep 08 18:38:16.987299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.987290    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl" event={"ID":"c0a1943b-6ef7-4645-8686-644057c96ea8","Type":"ContainerDied","Data":"5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68"}
Sep 08 18:38:16.987503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:16.987372    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fc45b867b34b497de757bfde9ebb09e9d6d15d7f8f6c29db01c48e3e0e61b68"
Sep 08 18:38:17.877873 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:17.877836    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"]
Sep 08 18:38:17.882013 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:17.881987    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad9a714c-m84kl"]
Sep 08 18:38:17.995080 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:17.995047    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:38:17.995335 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:17.995286    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy" containerID="cri-o://a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c" gracePeriod=30
Sep 08 18:38:18.101318 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:18.101283    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a1943b-6ef7-4645-8686-644057c96ea8" path="/var/lib/kubelet/pods/c0a1943b-6ef7-4645-8686-644057c96ea8/volumes"
Sep 08 18:38:23.063343 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.063311    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:23.063698 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.063684    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a1943b-6ef7-4645-8686-644057c96ea8" containerName="revoke-keys"
Sep 08 18:38:23.063776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.063699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a1943b-6ef7-4645-8686-644057c96ea8" containerName="revoke-keys"
Sep 08 18:38:23.063820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.063807    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a1943b-6ef7-4645-8686-644057c96ea8" containerName="revoke-keys"
Sep 08 18:38:23.065912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.065891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.069642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.069621    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-12c5e22e-dockercfg-hlgtz\""
Sep 08 18:38:23.069779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.069648    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-12c5e22e\""
Sep 08 18:38:23.076330 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.076308    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:23.089246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.089220    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:23.091918 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.091898    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.106979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.106952    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:23.115066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.115027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnpw\" (UniqueName: \"kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.115190 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.115078    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.115248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.115229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.115298 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.115280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8gr8\" (UniqueName: \"kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.216008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.215971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.216326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.216023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8gr8\" (UniqueName: \"kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.216326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.216055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnpw\" (UniqueName: \"kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.216326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.216081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.216795 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.216774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.216895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.216878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.225031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.225003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnpw\" (UniqueName: \"kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw\") pod \"payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.225135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.225074    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8gr8\" (UniqueName: \"kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8\") pod \"payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") " pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.270056 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.269993    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:23.316549 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zqbw\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316757 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316568    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316757 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316631    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316757 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316660    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316757 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316751    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316823    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.316978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.316854    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.317116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317048    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:38:23.317176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317145    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:38:23.317176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317169    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.317278 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:23.317633 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:38:23.317774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.317673    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:38:23.319031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.319007    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw" (OuterVolumeSpecName: "kube-api-access-6zqbw") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "kube-api-access-6zqbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:23.319579 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.319554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:38:23.319579 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.319558    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:23.390157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.390123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:23.404346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.404304    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:23.417671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417648    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy\") pod \"b68107a5-f972-4436-89ff-971e3fbd24ed\" (UID: \"b68107a5-f972-4436-89ff-971e3fbd24ed\") "
Sep 08 18:38:23.417921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417901    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.417921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417923    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.418078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417936    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.418078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417976    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.418078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.417990    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqbw\" (UniqueName: \"kubernetes.io/projected/b68107a5-f972-4436-89ff-971e3fbd24ed-kube-api-access-6zqbw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.418078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.418002    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68107a5-f972-4436-89ff-971e3fbd24ed-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.418078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.418014    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.419853 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.419827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b68107a5-f972-4436-89ff-971e3fbd24ed" (UID: "b68107a5-f972-4436-89ff-971e3fbd24ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:38:23.500286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.500253    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:38:23.500646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.500635    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.500711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.500649    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.500846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.500668    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.500897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.500852    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.501027 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.501012    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.501062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.501033    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerName="istio-proxy"
Sep 08 18:38:23.503698 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.503681    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.506507 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.506490    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-b3341dd9-kserve-self-signed-certs\""
Sep 08 18:38:23.516936 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.516909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:38:23.518588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.518762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.518762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.518762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t7gf\" (UniqueName: \"kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.518762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.518762 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.519040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.518777    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68107a5-f972-4436-89ff-971e3fbd24ed-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:23.619256 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t7gf\" (UniqueName: \"kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619256 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.619895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.619820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.622286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.622263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.622542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.622524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.629085 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.629061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t7gf\" (UniqueName: \"kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf\") pod \"e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") " pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:23.664344 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.664313    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:23.665850 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:23.665825    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb1d914_2698_43fd_abe6_2dd738d48911.slice/crio-ca86951b8ee9b83779e77e57ba6f806b05fe386e071412815f4ca581eacc8c10 WatchSource:0}: Error finding container ca86951b8ee9b83779e77e57ba6f806b05fe386e071412815f4ca581eacc8c10: Status 404 returned error can't find the container with id ca86951b8ee9b83779e77e57ba6f806b05fe386e071412815f4ca581eacc8c10
Sep 08 18:38:23.688324 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.688301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:23.697940 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:23.697911    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50cba197_a00f_4a70_9d61_0eea1e3a7ede.slice/crio-768522c701a685c5d8934a62e0bd85cff8a6f2b5033aad9598abe43e1648a24f WatchSource:0}: Error finding container 768522c701a685c5d8934a62e0bd85cff8a6f2b5033aad9598abe43e1648a24f: Status 404 returned error can't find the container with id 768522c701a685c5d8934a62e0bd85cff8a6f2b5033aad9598abe43e1648a24f
Sep 08 18:38:23.814523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:23.814485    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:24.028450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.028222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" event={"ID":"3bb1d914-2698-43fd-abe6-2dd738d48911","Type":"ContainerStarted","Data":"4db0261da5017f54712a658fafa21432a0f4bdc1071c93844e7c488055efeb78"}
Sep 08 18:38:24.028450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.028291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" event={"ID":"3bb1d914-2698-43fd-abe6-2dd738d48911","Type":"ContainerStarted","Data":"ca86951b8ee9b83779e77e57ba6f806b05fe386e071412815f4ca581eacc8c10"}
Sep 08 18:38:24.028450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.028323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:24.030463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.030435    2582 generic.go:356] "Generic (PLEG): container finished" podID="b68107a5-f972-4436-89ff-971e3fbd24ed" containerID="a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c" exitCode=0
Sep 08 18:38:24.030590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.030482    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"
Sep 08 18:38:24.030590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.030538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerDied","Data":"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"}
Sep 08 18:38:24.030590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.030585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5" event={"ID":"b68107a5-f972-4436-89ff-971e3fbd24ed","Type":"ContainerDied","Data":"dbf7593b77ebb102c6395ea80d6a4f5985c819684aad729949c356acc8656601"}
Sep 08 18:38:24.030844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.030618    2582 scope.go:117] "RemoveContainer" containerID="a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"
Sep 08 18:38:24.037135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.037076    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" event={"ID":"50cba197-a00f-4a70-9d61-0eea1e3a7ede","Type":"ContainerStarted","Data":"a130fde80c5ba4b9d179e61397c6b604503d57631f0a22d0143e811cc3616594"}
Sep 08 18:38:24.037221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.037138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" event={"ID":"50cba197-a00f-4a70-9d61-0eea1e3a7ede","Type":"ContainerStarted","Data":"768522c701a685c5d8934a62e0bd85cff8a6f2b5033aad9598abe43e1648a24f"}
Sep 08 18:38:24.037221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.037167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:24.045091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.045076    2582 scope.go:117] "RemoveContainer" containerID="5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"
Sep 08 18:38:24.052887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.052842    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" podStartSLOduration=1.052828055 podStartE2EDuration="1.052828055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:24.051592719 +0000 UTC m=+878.503167853" watchObservedRunningTime="2026-09-08 18:38:24.052828055 +0000 UTC m=+878.504403184"
Sep 08 18:38:24.053800 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.053776    2582 scope.go:117] "RemoveContainer" containerID="a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"
Sep 08 18:38:24.054113 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:38:24.054092    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c\": container with ID starting with a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c not found: ID does not exist" containerID="a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"
Sep 08 18:38:24.054184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.054121    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c"} err="failed to get container status \"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c\": rpc error: code = NotFound desc = could not find container \"a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c\": container with ID starting with a74e1d0937318f4647906102a71dfd635743736e349b9241d61f4c7532b5902c not found: ID does not exist"
Sep 08 18:38:24.054184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.054138    2582 scope.go:117] "RemoveContainer" containerID="5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"
Sep 08 18:38:24.054419 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:38:24.054403    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20\": container with ID starting with 5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20 not found: ID does not exist" containerID="5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"
Sep 08 18:38:24.054469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.054424    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20"} err="failed to get container status \"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20\": rpc error: code = NotFound desc = could not find container \"5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20\": container with ID starting with 5358917a9022ae2cb7790d030ef7a46ac70eaa0639d699d09fe3d171481ded20 not found: ID does not exist"
Sep 08 18:38:24.069933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.069882    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" podStartSLOduration=1.069864601 podStartE2EDuration="1.069864601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:24.069357082 +0000 UTC m=+878.520932214" watchObservedRunningTime="2026-09-08 18:38:24.069864601 +0000 UTC m=+878.521439734"
Sep 08 18:38:24.088157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.088100    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:38:24.091593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.091560    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:38:24.093231 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:24.093201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff91656c_49b6_4e42_8589_cfcd8f1b7844.slice/crio-53175ceb07b95cd49fa21c65c8dea1a87470886ef38493cf111cc9eeef84d6fd WatchSource:0}: Error finding container 53175ceb07b95cd49fa21c65c8dea1a87470886ef38493cf111cc9eeef84d6fd: Status 404 returned error can't find the container with id 53175ceb07b95cd49fa21c65c8dea1a87470886ef38493cf111cc9eeef84d6fd
Sep 08 18:38:24.099933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:24.099812    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ad9a714c-gw-openshift-default-68696ccbb4-5r2b5"]
Sep 08 18:38:25.045211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:25.045170    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" event={"ID":"ff91656c-49b6-4e42-8589-cfcd8f1b7844","Type":"ContainerStarted","Data":"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"}
Sep 08 18:38:25.045412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:25.045219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" event={"ID":"ff91656c-49b6-4e42-8589-cfcd8f1b7844","Type":"ContainerStarted","Data":"53175ceb07b95cd49fa21c65c8dea1a87470886ef38493cf111cc9eeef84d6fd"}
Sep 08 18:38:25.066566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:25.066503    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" podStartSLOduration=2.066475656 podStartE2EDuration="2.066475656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:25.062957065 +0000 UTC m=+879.514532198" watchObservedRunningTime="2026-09-08 18:38:25.066475656 +0000 UTC m=+879.518050789"
Sep 08 18:38:26.100318 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:26.100279    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b68107a5-f972-4436-89ff-971e3fbd24ed" path="/var/lib/kubelet/pods/b68107a5-f972-4436-89ff-971e3fbd24ed/volumes"
Sep 08 18:38:33.815503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:33.815466    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:33.816038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:33.815544    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:33.827913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:33.827891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:34.105439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:34.105357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:38:35.047442 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:35.047406    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:35.048050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:35.047493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:42.513112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.513034    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:38:42.516901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.516880    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.520770 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.520746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0eaf99\"/\"default-dockercfg-s6x9n\""
Sep 08 18:38:42.521820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.521799    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0eaf99\"/\"kube-root-ca.crt\""
Sep 08 18:38:42.522349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.522320    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0eaf99\"/\"autores-model-0eaf99-kserve-self-signed-certs\""
Sep 08 18:38:42.522656 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.522380    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0eaf99\"/\"openshift-service-ca.crt\""
Sep 08 18:38:42.529904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.529874    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:38:42.594831 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.594800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.594950 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.594876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.594950 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.594924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285m5\" (UniqueName: \"kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.595048 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.595026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.595116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.595098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.595156 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.595131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285m5\" (UniqueName: \"kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696607 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696607 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696754 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696710    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696860 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.696933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.696884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.698583 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.698564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.698982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.698967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.704632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.704604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-285m5\" (UniqueName: \"kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5\") pod \"autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") " pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:42.854801 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:42.854669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:43.128491 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:43.128372    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:38:43.131402 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:43.131372    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod591e011f_bc5f_416d_9ba3_931e6b87fb33.slice/crio-4ddc2640ebbc6f6b69d2ccfecf3fe48bfc11a03a7c91ee171ce07450b51287be WatchSource:0}: Error finding container 4ddc2640ebbc6f6b69d2ccfecf3fe48bfc11a03a7c91ee171ce07450b51287be: Status 404 returned error can't find the container with id 4ddc2640ebbc6f6b69d2ccfecf3fe48bfc11a03a7c91ee171ce07450b51287be
Sep 08 18:38:43.145497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:43.145469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" event={"ID":"591e011f-bc5f-416d-9ba3-931e6b87fb33","Type":"ContainerStarted","Data":"4ddc2640ebbc6f6b69d2ccfecf3fe48bfc11a03a7c91ee171ce07450b51287be"}
Sep 08 18:38:44.152604 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:44.152572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" event={"ID":"591e011f-bc5f-416d-9ba3-931e6b87fb33","Type":"ContainerStarted","Data":"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"}
Sep 08 18:38:44.173283 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:44.173222    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" podStartSLOduration=2.173207329 podStartE2EDuration="2.173207329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:44.172615757 +0000 UTC m=+898.624190890" watchObservedRunningTime="2026-09-08 18:38:44.173207329 +0000 UTC m=+898.624782498"
Sep 08 18:38:47.011872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.011834    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"]
Sep 08 18:38:47.018966 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.018946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:47.022144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.022121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:38:47.027088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.027062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"]
Sep 08 18:38:47.137176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.137127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6\" (UID: \"555728a7-2a4d-4710-8dd4-fbaaaa417746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:47.238642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.238605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6\" (UID: \"555728a7-2a4d-4710-8dd4-fbaaaa417746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:47.239335 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.239314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6\" (UID: \"555728a7-2a4d-4710-8dd4-fbaaaa417746\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:47.330347 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.330253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:47.605350 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:47.605320    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"]
Sep 08 18:38:47.607256 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:47.607208    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555728a7_2a4d_4710_8dd4_fbaaaa417746.slice/crio-e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128 WatchSource:0}: Error finding container e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128: Status 404 returned error can't find the container with id e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128
Sep 08 18:38:48.177427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:48.177386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6" event={"ID":"555728a7-2a4d-4710-8dd4-fbaaaa417746","Type":"ContainerStarted","Data":"46e82124d8408b2435423ef9007bf82e9e40c7f801a785f7e492f865eab11b75"}
Sep 08 18:38:48.177910 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:48.177454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6" event={"ID":"555728a7-2a4d-4710-8dd4-fbaaaa417746","Type":"ContainerStarted","Data":"e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128"}
Sep 08 18:38:48.196593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:48.196531    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6" podStartSLOduration=2.196509569 podStartE2EDuration="2.196509569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:48.19337475 +0000 UTC m=+902.644949879" watchObservedRunningTime="2026-09-08 18:38:48.196509569 +0000 UTC m=+902.648084701"
Sep 08 18:38:49.192088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:49.192049    2582 generic.go:356] "Generic (PLEG): container finished" podID="555728a7-2a4d-4710-8dd4-fbaaaa417746" containerID="46e82124d8408b2435423ef9007bf82e9e40c7f801a785f7e492f865eab11b75" exitCode=0
Sep 08 18:38:49.192466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:49.192137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6" event={"ID":"555728a7-2a4d-4710-8dd4-fbaaaa417746","Type":"ContainerDied","Data":"46e82124d8408b2435423ef9007bf82e9e40c7f801a785f7e492f865eab11b75"}
Sep 08 18:38:50.337675 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:50.337649    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:50.468601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:50.468519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca\") pod \"555728a7-2a4d-4710-8dd4-fbaaaa417746\" (UID: \"555728a7-2a4d-4710-8dd4-fbaaaa417746\") "
Sep 08 18:38:50.468883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:50.468860    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "555728a7-2a4d-4710-8dd4-fbaaaa417746" (UID: "555728a7-2a4d-4710-8dd4-fbaaaa417746"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:50.569942 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:50.569906    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555728a7-2a4d-4710-8dd4-fbaaaa417746-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:51.204338 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:51.204296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6" event={"ID":"555728a7-2a4d-4710-8dd4-fbaaaa417746","Type":"ContainerDied","Data":"e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128"}
Sep 08 18:38:51.204338 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:51.204316    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"
Sep 08 18:38:51.204338 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:51.204345    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6addf493b627b7baeb439bb249b747ff4cb60bc4aeb62a8bf73eca64fa50128"
Sep 08 18:38:52.855071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.855024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:52.855573 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.855089    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:52.867271 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.867244    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:52.969101 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.969069    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:52.969362 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.969326    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" podUID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" containerName="payload-processing" containerID="cri-o://a130fde80c5ba4b9d179e61397c6b604503d57631f0a22d0143e811cc3616594" gracePeriod=30
Sep 08 18:38:52.976427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.976400    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:52.976612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:52.976593    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" podUID="3bb1d914-2698-43fd-abe6-2dd738d48911" containerName="payload-pre-processing" containerID="cri-o://4db0261da5017f54712a658fafa21432a0f4bdc1071c93844e7c488055efeb78" gracePeriod=30
Sep 08 18:38:53.128784 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.128659    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"]
Sep 08 18:38:53.131045 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.131023    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12c5e22e-b25s6"]
Sep 08 18:38:53.229758 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.226375    2582 generic.go:356] "Generic (PLEG): container finished" podID="3bb1d914-2698-43fd-abe6-2dd738d48911" containerID="4db0261da5017f54712a658fafa21432a0f4bdc1071c93844e7c488055efeb78" exitCode=0
Sep 08 18:38:53.229758 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.226505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" event={"ID":"3bb1d914-2698-43fd-abe6-2dd738d48911","Type":"ContainerDied","Data":"4db0261da5017f54712a658fafa21432a0f4bdc1071c93844e7c488055efeb78"}
Sep 08 18:38:53.229758 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.228550    2582 generic.go:356] "Generic (PLEG): container finished" podID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" containerID="a130fde80c5ba4b9d179e61397c6b604503d57631f0a22d0143e811cc3616594" exitCode=0
Sep 08 18:38:53.229758 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.229597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" event={"ID":"50cba197-a00f-4a70-9d61-0eea1e3a7ede","Type":"ContainerDied","Data":"a130fde80c5ba4b9d179e61397c6b604503d57631f0a22d0143e811cc3616594"}
Sep 08 18:38:53.244575 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.244543    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:38:53.274194 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.274170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:53.278116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.278032    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:53.396240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396205    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hnpw\" (UniqueName: \"kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw\") pod \"3bb1d914-2698-43fd-abe6-2dd738d48911\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") "
Sep 08 18:38:53.396398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume\") pod \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") "
Sep 08 18:38:53.396398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8gr8\" (UniqueName: \"kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8\") pod \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\" (UID: \"50cba197-a00f-4a70-9d61-0eea1e3a7ede\") "
Sep 08 18:38:53.396482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume\") pod \"3bb1d914-2698-43fd-abe6-2dd738d48911\" (UID: \"3bb1d914-2698-43fd-abe6-2dd738d48911\") "
Sep 08 18:38:53.396707 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396597    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50cba197-a00f-4a70-9d61-0eea1e3a7ede" (UID: "50cba197-a00f-4a70-9d61-0eea1e3a7ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:53.396826 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.396789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bb1d914-2698-43fd-abe6-2dd738d48911" (UID: "3bb1d914-2698-43fd-abe6-2dd738d48911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:38:53.398591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.398567    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw" (OuterVolumeSpecName: "kube-api-access-7hnpw") pod "3bb1d914-2698-43fd-abe6-2dd738d48911" (UID: "3bb1d914-2698-43fd-abe6-2dd738d48911"). InnerVolumeSpecName "kube-api-access-7hnpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:53.398703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.398642    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8" (OuterVolumeSpecName: "kube-api-access-v8gr8") pod "50cba197-a00f-4a70-9d61-0eea1e3a7ede" (UID: "50cba197-a00f-4a70-9d61-0eea1e3a7ede"). InnerVolumeSpecName "kube-api-access-v8gr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:38:53.497587 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.497554    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hnpw\" (UniqueName: \"kubernetes.io/projected/3bb1d914-2698-43fd-abe6-2dd738d48911-kube-api-access-7hnpw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.497587 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.497583    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50cba197-a00f-4a70-9d61-0eea1e3a7ede-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.497587 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.497593    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8gr8\" (UniqueName: \"kubernetes.io/projected/50cba197-a00f-4a70-9d61-0eea1e3a7ede-kube-api-access-v8gr8\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:53.497829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:53.497601    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb1d914-2698-43fd-abe6-2dd738d48911-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:38:54.099347 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.099316    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555728a7-2a4d-4710-8dd4-fbaaaa417746" path="/var/lib/kubelet/pods/555728a7-2a4d-4710-8dd4-fbaaaa417746/volumes"
Sep 08 18:38:54.235782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.235753    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"
Sep 08 18:38:54.236024 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.235770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc" event={"ID":"50cba197-a00f-4a70-9d61-0eea1e3a7ede","Type":"ContainerDied","Data":"768522c701a685c5d8934a62e0bd85cff8a6f2b5033aad9598abe43e1648a24f"}
Sep 08 18:38:54.236178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.236163    2582 scope.go:117] "RemoveContainer" containerID="a130fde80c5ba4b9d179e61397c6b604503d57631f0a22d0143e811cc3616594"
Sep 08 18:38:54.238328 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.238289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q" event={"ID":"3bb1d914-2698-43fd-abe6-2dd738d48911","Type":"ContainerDied","Data":"ca86951b8ee9b83779e77e57ba6f806b05fe386e071412815f4ca581eacc8c10"}
Sep 08 18:38:54.238511 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.238306    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"
Sep 08 18:38:54.248716 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.248686    2582 scope.go:117] "RemoveContainer" containerID="4db0261da5017f54712a658fafa21432a0f4bdc1071c93844e7c488055efeb78"
Sep 08 18:38:54.262856 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.262825    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:54.268363 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.268337    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-12c5e22e-668587448f-8wbdc"]
Sep 08 18:38:54.280037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.280012    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:54.286896 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.286874    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-12c5e22e-5cbd7f7c86-sqb2q"]
Sep 08 18:38:54.953563 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.953514    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:38:54.954014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954000    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb1d914-2698-43fd-abe6-2dd738d48911" containerName="payload-pre-processing"
Sep 08 18:38:54.954074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954018    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb1d914-2698-43fd-abe6-2dd738d48911" containerName="payload-pre-processing"
Sep 08 18:38:54.954074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954044    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555728a7-2a4d-4710-8dd4-fbaaaa417746" containerName="revoke-keys"
Sep 08 18:38:54.954074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954050    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="555728a7-2a4d-4710-8dd4-fbaaaa417746" containerName="revoke-keys"
Sep 08 18:38:54.954074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954061    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" containerName="payload-processing"
Sep 08 18:38:54.954074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954068    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" containerName="payload-processing"
Sep 08 18:38:54.954285 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954131    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" containerName="payload-processing"
Sep 08 18:38:54.954285 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954143    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb1d914-2698-43fd-abe6-2dd738d48911" containerName="payload-pre-processing"
Sep 08 18:38:54.954285 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.954150    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="555728a7-2a4d-4710-8dd4-fbaaaa417746" containerName="revoke-keys"
Sep 08 18:38:54.958926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.958902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:54.961822 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.961793    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-18913a0c-openshift-default-dockercfg-7cmsl\""
Sep 08 18:38:54.967476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:54.967454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:38:55.113370 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113330    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxh7\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.113868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.113833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214651 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214651 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxh7\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214815    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.214928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.214929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.215275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.215241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.215412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.215382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.215701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.215445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.215701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.215635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.215840 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.215807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.217248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.217226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.217397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.217380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.225431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.225407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.226334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.226314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxh7\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7\") pod \"e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") " pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.271322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.271292    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:55.547691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.547664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:38:55.549103 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:38:55.549076    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4a72ca_e491_485f_a8c2_d2269898c946.slice/crio-0c6f987849b60bb48fabef8444cb33206d1dae220b1e872159470061452e5063 WatchSource:0}: Error finding container 0c6f987849b60bb48fabef8444cb33206d1dae220b1e872159470061452e5063: Status 404 returned error can't find the container with id 0c6f987849b60bb48fabef8444cb33206d1dae220b1e872159470061452e5063
Sep 08 18:38:55.551273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.551239    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:55.551345 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.551306    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:55.551345 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:55.551332    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:38:56.100191 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.100158    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb1d914-2698-43fd-abe6-2dd738d48911" path="/var/lib/kubelet/pods/3bb1d914-2698-43fd-abe6-2dd738d48911/volumes"
Sep 08 18:38:56.100526 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.100514    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cba197-a00f-4a70-9d61-0eea1e3a7ede" path="/var/lib/kubelet/pods/50cba197-a00f-4a70-9d61-0eea1e3a7ede/volumes"
Sep 08 18:38:56.252019 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.251984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerStarted","Data":"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"}
Sep 08 18:38:56.252354 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.252036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerStarted","Data":"0c6f987849b60bb48fabef8444cb33206d1dae220b1e872159470061452e5063"}
Sep 08 18:38:56.271542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.271512    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:56.272020 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.271983    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podStartSLOduration=2.271970885 podStartE2EDuration="2.271970885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:56.271715086 +0000 UTC m=+910.723290219" watchObservedRunningTime="2026-09-08 18:38:56.271970885 +0000 UTC m=+910.723546016"
Sep 08 18:38:56.277108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:56.277086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:57.257615 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:57.257581    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:38:57.258604 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:38:57.258585    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:14.771973 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.771928    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"]
Sep 08 18:39:14.776057 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.776025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:14.786751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.783028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:39:14.791028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.790996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"]
Sep 08 18:39:14.813079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.813038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr\" (UID: \"2b94c39a-e8d5-48e1-926f-20845e09c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:14.914598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.914553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr\" (UID: \"2b94c39a-e8d5-48e1-926f-20845e09c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:14.915209 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:14.915189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr\" (UID: \"2b94c39a-e8d5-48e1-926f-20845e09c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:15.100388 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:15.100288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:15.374932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:15.374841    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"]
Sep 08 18:39:15.382937 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:39:15.382894    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b94c39a_e8d5_48e1_926f_20845e09c1df.slice/crio-8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321 WatchSource:0}: Error finding container 8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321: Status 404 returned error can't find the container with id 8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321
Sep 08 18:39:15.533825 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:15.533796    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:39:16.367357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:16.367323    2582 generic.go:356] "Generic (PLEG): container finished" podID="2b94c39a-e8d5-48e1-926f-20845e09c1df" containerID="eee451810923d87a6f051113a46c68725c9a7c6af6ca1bfbebba7a5624dafda6" exitCode=0
Sep 08 18:39:16.367749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:16.367420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr" event={"ID":"2b94c39a-e8d5-48e1-926f-20845e09c1df","Type":"ContainerDied","Data":"eee451810923d87a6f051113a46c68725c9a7c6af6ca1bfbebba7a5624dafda6"}
Sep 08 18:39:16.367749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:16.367489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr" event={"ID":"2b94c39a-e8d5-48e1-926f-20845e09c1df","Type":"ContainerStarted","Data":"8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321"}
Sep 08 18:39:16.396871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:16.396827    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:39:17.521677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:17.521654    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:17.641591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:17.641554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca\") pod \"2b94c39a-e8d5-48e1-926f-20845e09c1df\" (UID: \"2b94c39a-e8d5-48e1-926f-20845e09c1df\") "
Sep 08 18:39:17.641951 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:17.641930    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b94c39a-e8d5-48e1-926f-20845e09c1df" (UID: "2b94c39a-e8d5-48e1-926f-20845e09c1df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:17.642026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:17.642014    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b94c39a-e8d5-48e1-926f-20845e09c1df-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:18.379627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:18.379553    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"
Sep 08 18:39:18.379627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:18.379574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr" event={"ID":"2b94c39a-e8d5-48e1-926f-20845e09c1df","Type":"ContainerDied","Data":"8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321"}
Sep 08 18:39:18.379837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:18.379631    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7b3b994d538f40f768aae0dc8f78cf88ff376dbd8af50d84af9428ec800321"
Sep 08 18:39:18.511685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:18.511647    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:39:19.386026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.385998    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerID="3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7" exitCode=0
Sep 08 18:39:19.386514 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.386077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerDied","Data":"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"}
Sep 08 18:39:19.386645 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.386627    2582 scope.go:117] "RemoveContainer" containerID="3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"
Sep 08 18:39:19.387741 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.387695    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:19.387842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.387793    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:19.387842 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:19.387825    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:20.393610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:20.393576    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerStarted","Data":"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"}
Sep 08 18:39:21.271446 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:21.271403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:21.271717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:21.271696    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:21.271800 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:21.271763    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:22.272426 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.272385    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:22.273075 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.273042    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:22.444958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.444922    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"]
Sep 08 18:39:22.445333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.445320    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b94c39a-e8d5-48e1-926f-20845e09c1df" containerName="revoke-keys"
Sep 08 18:39:22.445377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.445334    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b94c39a-e8d5-48e1-926f-20845e09c1df" containerName="revoke-keys"
Sep 08 18:39:22.445411 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.445399    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b94c39a-e8d5-48e1-926f-20845e09c1df" containerName="revoke-keys"
Sep 08 18:39:22.448625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.448609    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:22.451469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.451444    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:39:22.458690 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.458658    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"]
Sep 08 18:39:22.489617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.489510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv\" (UID: \"ac118867-6aee-40ff-b12e-b37699eb3e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:22.591149 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.591042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv\" (UID: \"ac118867-6aee-40ff-b12e-b37699eb3e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:22.591670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.591650    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv\" (UID: \"ac118867-6aee-40ff-b12e-b37699eb3e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:22.760429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:22.760385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:23.043690 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:23.043655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"]
Sep 08 18:39:23.046336 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:39:23.046309    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac118867_6aee_40ff_b12e_b37699eb3e82.slice/crio-684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b WatchSource:0}: Error finding container 684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b: Status 404 returned error can't find the container with id 684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b
Sep 08 18:39:23.272098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:23.272067    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:23.272344 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:23.272298    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:23.411636 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:23.411603    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv" event={"ID":"ac118867-6aee-40ff-b12e-b37699eb3e82","Type":"ContainerStarted","Data":"684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b"}
Sep 08 18:39:24.272341 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:24.272295    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:24.272542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:24.272379    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:24.418223 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:24.418181    2582 generic.go:356] "Generic (PLEG): container finished" podID="ac118867-6aee-40ff-b12e-b37699eb3e82" containerID="433fe10f0c4dae3b5c454729fa13b9cf24c0373c56c441c994cca5a80e7d6c4d" exitCode=0
Sep 08 18:39:24.418616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:24.418221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv" event={"ID":"ac118867-6aee-40ff-b12e-b37699eb3e82","Type":"ContainerDied","Data":"433fe10f0c4dae3b5c454729fa13b9cf24c0373c56c441c994cca5a80e7d6c4d"}
Sep 08 18:39:25.272263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.272226    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:25.272550 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.272523    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:25.593805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.592889    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:39:25.593805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.593154    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" podUID="166a2a81-d758-44a3-88df-4a5ba56eb677" containerName="payload-pre-processing" containerID="cri-o://cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c" gracePeriod=30
Sep 08 18:39:25.685091 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.685059    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:25.722956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.722924    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca\") pod \"ac118867-6aee-40ff-b12e-b37699eb3e82\" (UID: \"ac118867-6aee-40ff-b12e-b37699eb3e82\") "
Sep 08 18:39:25.723303 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.723265    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac118867-6aee-40ff-b12e-b37699eb3e82" (UID: "ac118867-6aee-40ff-b12e-b37699eb3e82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:25.743476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.743411    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"]
Sep 08 18:39:25.746336 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.746306    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0eaf99-qhvpr"]
Sep 08 18:39:25.824556 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.824524    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac118867-6aee-40ff-b12e-b37699eb3e82-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:25.893806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:25.893779    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:39:26.026972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.026935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume\") pod \"166a2a81-d758-44a3-88df-4a5ba56eb677\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") "
Sep 08 18:39:26.027159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.026992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l9bk\" (UniqueName: \"kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk\") pod \"166a2a81-d758-44a3-88df-4a5ba56eb677\" (UID: \"166a2a81-d758-44a3-88df-4a5ba56eb677\") "
Sep 08 18:39:26.027316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.027294    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "166a2a81-d758-44a3-88df-4a5ba56eb677" (UID: "166a2a81-d758-44a3-88df-4a5ba56eb677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:26.029205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.029175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk" (OuterVolumeSpecName: "kube-api-access-9l9bk") pod "166a2a81-d758-44a3-88df-4a5ba56eb677" (UID: "166a2a81-d758-44a3-88df-4a5ba56eb677"). InnerVolumeSpecName "kube-api-access-9l9bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:26.100608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.100576    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b94c39a-e8d5-48e1-926f-20845e09c1df" path="/var/lib/kubelet/pods/2b94c39a-e8d5-48e1-926f-20845e09c1df/volumes"
Sep 08 18:39:26.128144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.128116    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/166a2a81-d758-44a3-88df-4a5ba56eb677-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:26.128144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.128141    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9bk\" (UniqueName: \"kubernetes.io/projected/166a2a81-d758-44a3-88df-4a5ba56eb677-kube-api-access-9l9bk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:26.272125 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.272084    2582 patch_prober.go:28] interesting pod/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 08 18:39:26.272332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.272161    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 08 18:39:26.429957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.429926    2582 generic.go:356] "Generic (PLEG): container finished" podID="166a2a81-d758-44a3-88df-4a5ba56eb677" containerID="cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c" exitCode=0
Sep 08 18:39:26.430151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.429977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"
Sep 08 18:39:26.430151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.430023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" event={"ID":"166a2a81-d758-44a3-88df-4a5ba56eb677","Type":"ContainerDied","Data":"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"}
Sep 08 18:39:26.430151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.430085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx" event={"ID":"166a2a81-d758-44a3-88df-4a5ba56eb677","Type":"ContainerDied","Data":"7d5847c9a68eadda249d0271069bdc055e4d93e48a5da48b92d359d931158f08"}
Sep 08 18:39:26.430151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.430113    2582 scope.go:117] "RemoveContainer" containerID="cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"
Sep 08 18:39:26.431627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.431463    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"
Sep 08 18:39:26.431627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.431476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv" event={"ID":"ac118867-6aee-40ff-b12e-b37699eb3e82","Type":"ContainerDied","Data":"684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b"}
Sep 08 18:39:26.431627 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.431517    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="684a7e2fd47f8f9ab0a987c87c66890d6f7e09a7f6e64ffde2299cb705f3f66b"
Sep 08 18:39:26.439901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.439883    2582 scope.go:117] "RemoveContainer" containerID="cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"
Sep 08 18:39:26.440218 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:26.440198    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c\": container with ID starting with cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c not found: ID does not exist" containerID="cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"
Sep 08 18:39:26.440287 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.440229    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c"} err="failed to get container status \"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c\": rpc error: code = NotFound desc = could not find container \"cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c\": container with ID starting with cda52d7f55f685fe89b9dfc83bc8e8fb5b5f21f709700344b73a1b25c954310c not found: ID does not exist"
Sep 08 18:39:26.448998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.448976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:39:26.454783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:26.454761    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0eaf99-66bd5ffff4-q5rjx"]
Sep 08 18:39:27.257509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:27.257474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:27.275458 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:27.275429    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:27.439331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:27.439301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:28.099824 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:28.099787    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166a2a81-d758-44a3-88df-4a5ba56eb677" path="/var/lib/kubelet/pods/166a2a81-d758-44a3-88df-4a5ba56eb677/volumes"
Sep 08 18:39:30.360368 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.360325    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"]
Sep 08 18:39:30.363282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.363254    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18913a0c-xfbgv"]
Sep 08 18:39:30.491542 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.491503    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:39:30.491779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.491713    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy" containerID="cri-o://568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992" gracePeriod=30
Sep 08 18:39:30.553769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.553711    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" podUID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" containerName="main" containerID="cri-o://7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3" gracePeriod=45
Sep 08 18:39:30.818076 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.818047    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:39:30.870788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870756    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.870926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870801    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.870926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.870926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.870926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870926    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t7gf\" (UniqueName: \"kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.871155 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.870977    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache\") pod \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\" (UID: \"ff91656c-49b6-4e42-8589-cfcd8f1b7844\") "
Sep 08 18:39:30.871208 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.871175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:30.871269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.871201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home" (OuterVolumeSpecName: "home") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:30.871369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.871317    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.871423 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.871375    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.871423 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.871392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:30.873311 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.873279    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:30.873459 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.873436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm" (OuterVolumeSpecName: "dshm") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:30.873508 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.873441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf" (OuterVolumeSpecName: "kube-api-access-7t7gf") pod "ff91656c-49b6-4e42-8589-cfcd8f1b7844" (UID: "ff91656c-49b6-4e42-8589-cfcd8f1b7844"). InnerVolumeSpecName "kube-api-access-7t7gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:30.972658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.972560    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t7gf\" (UniqueName: \"kubernetes.io/projected/ff91656c-49b6-4e42-8589-cfcd8f1b7844-kube-api-access-7t7gf\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.972658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.972610    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.972658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.972626    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff91656c-49b6-4e42-8589-cfcd8f1b7844-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:30.972658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:30.972638    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff91656c-49b6-4e42-8589-cfcd8f1b7844-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.418352 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.418293    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" podUID="591e011f-bc5f-416d-9ba3-931e6b87fb33" containerName="main" containerID="cri-o://c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c" gracePeriod=45
Sep 08 18:39:31.461705 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.461670    2582 generic.go:356] "Generic (PLEG): container finished" podID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" containerID="7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3" exitCode=0
Sep 08 18:39:31.461887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.461776    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" event={"ID":"ff91656c-49b6-4e42-8589-cfcd8f1b7844","Type":"ContainerDied","Data":"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"}
Sep 08 18:39:31.461887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.461791    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"
Sep 08 18:39:31.461887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.461813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt" event={"ID":"ff91656c-49b6-4e42-8589-cfcd8f1b7844","Type":"ContainerDied","Data":"53175ceb07b95cd49fa21c65c8dea1a87470886ef38493cf111cc9eeef84d6fd"}
Sep 08 18:39:31.461887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.461834    2582 scope.go:117] "RemoveContainer" containerID="7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"
Sep 08 18:39:31.488378 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.488349    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:39:31.493625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.493596    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-b3341dd9-kserve-74769587fc-m9gjt"]
Sep 08 18:39:31.529913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.529761    2582 scope.go:117] "RemoveContainer" containerID="7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"
Sep 08 18:39:31.530198 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:31.530175    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3\": container with ID starting with 7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3 not found: ID does not exist" containerID="7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"
Sep 08 18:39:31.530281 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.530208    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3"} err="failed to get container status \"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3\": rpc error: code = NotFound desc = could not find container \"7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3\": container with ID starting with 7889ef6471cf304599757876ef30267cd19b488e8cb9ac8a7a1e72a5130ca2f3 not found: ID does not exist"
Sep 08 18:39:31.662253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.662226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:39:31.781058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.780970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-285m5\" (UniqueName: \"kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781299    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home" (OuterVolumeSpecName: "home") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:31.781455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir\") pod \"591e011f-bc5f-416d-9ba3-931e6b87fb33\" (UID: \"591e011f-bc5f-416d-9ba3-931e6b87fb33\") "
Sep 08 18:39:31.781455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache" (OuterVolumeSpecName: "model-cache") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:31.781576 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781491    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:31.781670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781655    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.781709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781672    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.781709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.781681    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.783468 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.783441    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5" (OuterVolumeSpecName: "kube-api-access-285m5") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "kube-api-access-285m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:31.783804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.783785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm" (OuterVolumeSpecName: "dshm") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:31.783804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.783796    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "591e011f-bc5f-416d-9ba3-931e6b87fb33" (UID: "591e011f-bc5f-416d-9ba3-931e6b87fb33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:31.882428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.882385    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285m5\" (UniqueName: \"kubernetes.io/projected/591e011f-bc5f-416d-9ba3-931e6b87fb33-kube-api-access-285m5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.882428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.882420    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591e011f-bc5f-416d-9ba3-931e6b87fb33-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:31.882428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:31.882435    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591e011f-bc5f-416d-9ba3-931e6b87fb33-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:32.098906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.098828    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac118867-6aee-40ff-b12e-b37699eb3e82" path="/var/lib/kubelet/pods/ac118867-6aee-40ff-b12e-b37699eb3e82/volumes"
Sep 08 18:39:32.099222 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.099208    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" path="/var/lib/kubelet/pods/ff91656c-49b6-4e42-8589-cfcd8f1b7844/volumes"
Sep 08 18:39:32.467983 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.467941    2582 generic.go:356] "Generic (PLEG): container finished" podID="591e011f-bc5f-416d-9ba3-931e6b87fb33" containerID="c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c" exitCode=0
Sep 08 18:39:32.468457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.468011    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"
Sep 08 18:39:32.468457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.468033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" event={"ID":"591e011f-bc5f-416d-9ba3-931e6b87fb33","Type":"ContainerDied","Data":"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"}
Sep 08 18:39:32.468457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.468086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf" event={"ID":"591e011f-bc5f-416d-9ba3-931e6b87fb33","Type":"ContainerDied","Data":"4ddc2640ebbc6f6b69d2ccfecf3fe48bfc11a03a7c91ee171ce07450b51287be"}
Sep 08 18:39:32.468457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.468107    2582 scope.go:117] "RemoveContainer" containerID="c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"
Sep 08 18:39:32.477307 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.477293    2582 scope.go:117] "RemoveContainer" containerID="c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"
Sep 08 18:39:32.477573 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:32.477557    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c\": container with ID starting with c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c not found: ID does not exist" containerID="c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"
Sep 08 18:39:32.477620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.477581    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c"} err="failed to get container status \"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c\": rpc error: code = NotFound desc = could not find container \"c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c\": container with ID starting with c5d14167b646ee84af9358ca32b15960ffc386db84512e163520c4fba8c8a77c not found: ID does not exist"
Sep 08 18:39:32.487117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.487095    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:39:32.493207 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:32.493187    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0eaf99/autores-model-0eaf99-kserve-6fdd9d47dc-jw7bf"]
Sep 08 18:39:33.532240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.532197    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerName="main" containerID="cri-o://524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9" gracePeriod=45
Sep 08 18:39:33.788063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.788005    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:39:33.900467 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900434    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900467 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900468    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900506    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d52q\" (UniqueName: \"kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900548    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache\") pod \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\" (UID: \"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7\") "
Sep 08 18:39:33.900975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900847    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:33.900975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home" (OuterVolumeSpecName: "home") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:33.900975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900948    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:33.901090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.900997    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:33.901090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.901010    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:33.902837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.902810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:39:33.902837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.902812    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm" (OuterVolumeSpecName: "dshm") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:33.902990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:33.902844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q" (OuterVolumeSpecName: "kube-api-access-7d52q") pod "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" (UID: "a061d5ab-0aec-4d05-ba32-1e6b7af7fed7"). InnerVolumeSpecName "kube-api-access-7d52q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:34.001706 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.001673    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:34.001706 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.001702    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:34.001706 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.001712    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d52q\" (UniqueName: \"kubernetes.io/projected/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-kube-api-access-7d52q\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:34.001959 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.001746    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:34.099698 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.099603    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591e011f-bc5f-416d-9ba3-931e6b87fb33" path="/var/lib/kubelet/pods/591e011f-bc5f-416d-9ba3-931e6b87fb33/volumes"
Sep 08 18:39:34.481555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.481518    2582 generic.go:356] "Generic (PLEG): container finished" podID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerID="524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9" exitCode=0
Sep 08 18:39:34.481703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.481566    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"
Sep 08 18:39:34.481703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.481607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" event={"ID":"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7","Type":"ContainerDied","Data":"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"}
Sep 08 18:39:34.481703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.481657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" event={"ID":"a061d5ab-0aec-4d05-ba32-1e6b7af7fed7","Type":"ContainerDied","Data":"390bd3ba9cc0bcd8435bbacda14ef33fb2fab36feb0f4658a0fa5ab8a5bee0cc"}
Sep 08 18:39:34.481703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.481677    2582 scope.go:117] "RemoveContainer" containerID="524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"
Sep 08 18:39:34.490565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.490545    2582 scope.go:117] "RemoveContainer" containerID="524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"
Sep 08 18:39:34.490976 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:34.490955    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9\": container with ID starting with 524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9 not found: ID does not exist" containerID="524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"
Sep 08 18:39:34.491037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.490985    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9"} err="failed to get container status \"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9\": rpc error: code = NotFound desc = could not find container \"524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9\": container with ID starting with 524ab25a4d87f16c32ebaad68166f06908b6c7640bb0ec6bdb586356e2e8f9b9 not found: ID does not exist"
Sep 08 18:39:34.499144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.499118    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:39:34.503370 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:34.503348    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd"]
Sep 08 18:39:35.748371 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.748346    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:35.818445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818362    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818433    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818465    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvxh7\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818572    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818596    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.818703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818687    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:35.819037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token\") pod \"4c4a72ca-e491-485f-a8c2-d2269898c946\" (UID: \"4c4a72ca-e491-485f-a8c2-d2269898c946\") "
Sep 08 18:39:35.819037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.818803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:35.819134 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:35.819134 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:35.819238 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819186    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4a72ca-e491-485f-a8c2-d2269898c946-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.819238 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819204    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.819238 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819219    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.819375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819236    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.819375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.819348    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:35.820870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.820844    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:39:35.820979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.820946    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7" (OuterVolumeSpecName: "kube-api-access-lvxh7") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "kube-api-access-lvxh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:35.821141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.821113    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:35.821141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.821123    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c4a72ca-e491-485f-a8c2-d2269898c946" (UID: "4c4a72ca-e491-485f-a8c2-d2269898c946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:35.919913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.919878    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.919913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.919908    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.919913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.919917    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxh7\" (UniqueName: \"kubernetes.io/projected/4c4a72ca-e491-485f-a8c2-d2269898c946-kube-api-access-lvxh7\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.920141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.919926    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:35.920141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:35.919934    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4a72ca-e491-485f-a8c2-d2269898c946-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:36.106704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.106621    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" path="/var/lib/kubelet/pods/a061d5ab-0aec-4d05-ba32-1e6b7af7fed7/volumes"
Sep 08 18:39:36.494437 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.494404    2582 generic.go:356] "Generic (PLEG): container finished" podID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerID="568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992" exitCode=0
Sep 08 18:39:36.494437 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.494441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerDied","Data":"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"}
Sep 08 18:39:36.494668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.494467    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"
Sep 08 18:39:36.494668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.494478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg" event={"ID":"4c4a72ca-e491-485f-a8c2-d2269898c946","Type":"ContainerDied","Data":"0c6f987849b60bb48fabef8444cb33206d1dae220b1e872159470061452e5063"}
Sep 08 18:39:36.494668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.494502    2582 scope.go:117] "RemoveContainer" containerID="568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"
Sep 08 18:39:36.503324 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.503308    2582 scope.go:117] "RemoveContainer" containerID="3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"
Sep 08 18:39:36.511465 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.511447    2582 scope.go:117] "RemoveContainer" containerID="568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"
Sep 08 18:39:36.511794 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:36.511743    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992\": container with ID starting with 568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992 not found: ID does not exist" containerID="568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"
Sep 08 18:39:36.511794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.511778    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992"} err="failed to get container status \"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992\": rpc error: code = NotFound desc = could not find container \"568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992\": container with ID starting with 568962efca67e567e1a972e597bb9206793055cf53ffb59ab7395dc5fd777992 not found: ID does not exist"
Sep 08 18:39:36.511964 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.511799    2582 scope.go:117] "RemoveContainer" containerID="3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"
Sep 08 18:39:36.512116 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:36.512097    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7\": container with ID starting with 3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7 not found: ID does not exist" containerID="3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"
Sep 08 18:39:36.512169 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.512122    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7"} err="failed to get container status \"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7\": rpc error: code = NotFound desc = could not find container \"3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7\": container with ID starting with 3b37caf731142c8f18b04e4296c0de8a9b8e7eea9c8534f94d91da0c66829fb7 not found: ID does not exist"
Sep 08 18:39:36.514898 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.514878    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:39:36.518675 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:36.518653    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-18913a0c-openshift-default-7b566fffc6-8nshg"]
Sep 08 18:39:37.265556 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:37.265525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:39:37.266014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:37.265781    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy" containerID="cri-o://97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd" gracePeriod=30
Sep 08 18:39:38.004001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.003969    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:39:38.004538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004522    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac118867-6aee-40ff-b12e-b37699eb3e82" containerName="revoke-keys"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004542    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac118867-6aee-40ff-b12e-b37699eb3e82" containerName="revoke-keys"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004558    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591e011f-bc5f-416d-9ba3-931e6b87fb33" containerName="main"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004567    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="591e011f-bc5f-416d-9ba3-931e6b87fb33" containerName="main"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004577    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004585    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004597    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166a2a81-d758-44a3-88df-4a5ba56eb677" containerName="payload-pre-processing"
Sep 08 18:39:38.004616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004605    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="166a2a81-d758-44a3-88df-4a5ba56eb677" containerName="payload-pre-processing"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004619    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004627    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004649    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004657    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004669    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004677    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004795    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac118867-6aee-40ff-b12e-b37699eb3e82" containerName="revoke-keys"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004808    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="166a2a81-d758-44a3-88df-4a5ba56eb677" containerName="payload-pre-processing"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004819    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff91656c-49b6-4e42-8589-cfcd8f1b7844" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004828    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004837    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="591e011f-bc5f-416d-9ba3-931e6b87fb33" containerName="main"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004846    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.004998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.004854    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" containerName="istio-proxy"
Sep 08 18:39:38.009271 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.009251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.012356 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.012332    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c5ee49-openshift-default-dockercfg-tq48w\""
Sep 08 18:39:38.039746 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.023318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:39:38.099866 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.099832    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c4a72ca-e491-485f-a8c2-d2269898c946" path="/var/lib/kubelet/pods/4c4a72ca-e491-485f-a8c2-d2269898c946/volumes"
Sep 08 18:39:38.149441 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149441 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149442    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nmk\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149927 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.149927 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.149789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250330 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nmk\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250739 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.250972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.250894    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251510    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251438    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.251662 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.251384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.253404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.253384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.253483 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.253426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.259466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.259411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nmk\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.259782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.259719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token\") pod \"e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") " pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.348169 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.348135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:38.629761 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.629711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:39:38.632272 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:39:38.632235    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323c3360_4481_42d4_9e13_2aa74fa37e22.slice/crio-64c0ffa4db18680d5fe816d327750165177c4d79570747ee0e453221febd6a77 WatchSource:0}: Error finding container 64c0ffa4db18680d5fe816d327750165177c4d79570747ee0e453221febd6a77: Status 404 returned error can't find the container with id 64c0ffa4db18680d5fe816d327750165177c4d79570747ee0e453221febd6a77
Sep 08 18:39:38.634738 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.634691    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:38.634815 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.634797    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:38.634862 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.634844    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:39:38.695818 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:38.695711    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-conflict-a-b3341dd9-kserve-7f48fcd467-tm7dd" podUID="a061d5ab-0aec-4d05-ba32-1e6b7af7fed7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 18:39:39.513136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:39.513100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerStarted","Data":"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"}
Sep 08 18:39:39.513519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:39.513150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerStarted","Data":"64c0ffa4db18680d5fe816d327750165177c4d79570747ee0e453221febd6a77"}
Sep 08 18:39:39.537686 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:39.537641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podStartSLOduration=2.53762495 podStartE2EDuration="2.53762495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:39.535670586 +0000 UTC m=+953.987245717" watchObservedRunningTime="2026-09-08 18:39:39.53762495 +0000 UTC m=+953.989200129"
Sep 08 18:39:40.349143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:40.349104    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:40.354415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:40.354391    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:40.518792 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:40.518746    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:40.519817 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:40.519797    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:39:42.515978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.515954    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:39:42.530920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.530816    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerID="97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd" exitCode=0
Sep 08 18:39:42.530920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.530865    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"
Sep 08 18:39:42.531160 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.530918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerDied","Data":"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"}
Sep 08 18:39:42.531160 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.531002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng" event={"ID":"b7687c06-8b91-41d0-95f8-42fdb9be3d53","Type":"ContainerDied","Data":"066de7701bc083664af07474346261656f2d6a8838779e057b9a3cbf6454ae23"}
Sep 08 18:39:42.531160 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.531058    2582 scope.go:117] "RemoveContainer" containerID="97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"
Sep 08 18:39:42.541697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.541672    2582 scope.go:117] "RemoveContainer" containerID="bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"
Sep 08 18:39:42.554415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.554395    2582 scope.go:117] "RemoveContainer" containerID="97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"
Sep 08 18:39:42.554778 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:42.554755    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd\": container with ID starting with 97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd not found: ID does not exist" containerID="97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"
Sep 08 18:39:42.554883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.554792    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd"} err="failed to get container status \"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd\": rpc error: code = NotFound desc = could not find container \"97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd\": container with ID starting with 97c922b41354b3fd563d4c17e10898f638b6ef2700548f2c6453d17d05f60dcd not found: ID does not exist"
Sep 08 18:39:42.554883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.554817    2582 scope.go:117] "RemoveContainer" containerID="bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"
Sep 08 18:39:42.555119 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:39:42.555100    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6\": container with ID starting with bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6 not found: ID does not exist" containerID="bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"
Sep 08 18:39:42.555189 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.555127    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6"} err="failed to get container status \"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6\": rpc error: code = NotFound desc = could not find container \"bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6\": container with ID starting with bc8cdc3bc37bfd81b20edea2f0a393a0f1ef03e976013d91ec566b086c9d67c6 not found: ID does not exist"
Sep 08 18:39:42.697102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697281    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2zjl\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697369    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697481    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697560    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket\") pod \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\" (UID: \"b7687c06-8b91-41d0-95f8-42fdb9be3d53\") "
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:42.697618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697577    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:42.698031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.697820    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:42.698031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.698006    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.698031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.698025    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.698295 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.698040    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.698295 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.698055    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.698295 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.698067    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.699986 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.699954    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:39:42.700155 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.700132    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:42.700496 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.700476    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:39:42.700560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.700534    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl" (OuterVolumeSpecName: "kube-api-access-w2zjl") pod "b7687c06-8b91-41d0-95f8-42fdb9be3d53" (UID: "b7687c06-8b91-41d0-95f8-42fdb9be3d53"). InnerVolumeSpecName "kube-api-access-w2zjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:39:42.799517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.799423    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.799517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.799454    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.799517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.799466    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.799517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.799477    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zjl\" (UniqueName: \"kubernetes.io/projected/b7687c06-8b91-41d0-95f8-42fdb9be3d53-kube-api-access-w2zjl\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:39:42.854992 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.854963    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:39:42.859562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:42.859535    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0eaf99-openshift-default-6885f8956b-g4xng"]
Sep 08 18:39:43.207766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.207718    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:39:43.208133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.208121    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:39:43.208177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.208136    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:39:43.208241 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.208200    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:39:43.208241 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.208209    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:39:43.212539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.212517    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.215246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.215223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c5ee49-dockercfg-rkblk\""
Sep 08 18:39:43.215246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.215237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c5ee49\""
Sep 08 18:39:43.220437 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.220412    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:39:43.304172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.304137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhj8q\" (UniqueName: \"kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.304395 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.304282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.404996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.404958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.405144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.405024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhj8q\" (UniqueName: \"kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.405631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.405613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.414132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.414109    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhj8q\" (UniqueName: \"kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q\") pod \"payload-processing-e2e-explicit-c5ee49-769f8766f-974kf\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") " pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.524554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.524453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:43.796323 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:43.796298    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:39:43.798366 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:39:43.798326    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26d186b_93dc_4441_a2ba_f1c547aab42b.slice/crio-2e76e5a44ba1f1e962e9869c87aa38e235053f5b6d6d9275a986dd032a47ef8d WatchSource:0}: Error finding container 2e76e5a44ba1f1e962e9869c87aa38e235053f5b6d6d9275a986dd032a47ef8d: Status 404 returned error can't find the container with id 2e76e5a44ba1f1e962e9869c87aa38e235053f5b6d6d9275a986dd032a47ef8d
Sep 08 18:39:44.103713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:44.103610    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" path="/var/lib/kubelet/pods/b7687c06-8b91-41d0-95f8-42fdb9be3d53/volumes"
Sep 08 18:39:44.545230 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:44.545194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" event={"ID":"a26d186b-93dc-4441-a2ba-f1c547aab42b","Type":"ContainerStarted","Data":"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"}
Sep 08 18:39:44.545712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:44.545253    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" event={"ID":"a26d186b-93dc-4441-a2ba-f1c547aab42b","Type":"ContainerStarted","Data":"2e76e5a44ba1f1e962e9869c87aa38e235053f5b6d6d9275a986dd032a47ef8d"}
Sep 08 18:39:44.545712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:44.545381    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:39:44.563905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:44.563855    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" podStartSLOduration=1.563838798 podStartE2EDuration="1.563838798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:44.562016086 +0000 UTC m=+959.013591220" watchObservedRunningTime="2026-09-08 18:39:44.563838798 +0000 UTC m=+959.015413930"
Sep 08 18:39:55.551920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:39:55.551887    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:40:01.637428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.637395    2582 generic.go:356] "Generic (PLEG): container finished" podID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerID="eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3" exitCode=0
Sep 08 18:40:01.637877 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.637477    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerDied","Data":"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"}
Sep 08 18:40:01.637924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.637895    2582 scope.go:117] "RemoveContainer" containerID="eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"
Sep 08 18:40:01.639110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.639084    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:40:01.639185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.639150    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:40:01.639185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:01.639181    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:40:02.644765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:02.644712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerStarted","Data":"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"}
Sep 08 18:40:03.348745 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:03.348692    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:40:03.348944 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:03.348922    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:03.349000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:03.348982    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:04.349030 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.348996    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:04.349434 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.349064    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:04.641114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.641073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:40:04.642069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.642040    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:40:04.642069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.642068    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7687c06-8b91-41d0-95f8-42fdb9be3d53" containerName="istio-proxy"
Sep 08 18:40:04.645039 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.645011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.648488 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.648468    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15d796\"/\"kube-root-ca.crt\""
Sep 08 18:40:04.650158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.650135    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15d796\"/\"openshift-service-ca.crt\""
Sep 08 18:40:04.650282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.650183    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15d796\"/\"default-dockercfg-zjjc8\""
Sep 08 18:40:04.650677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.650582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15d796\"/\"test-model-15d796-kserve-self-signed-certs\""
Sep 08 18:40:04.653858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.653655    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:40:04.701551 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.701750 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.701750 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hndz\" (UniqueName: \"kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.701750 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.701879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701767    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.701879 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.701790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.802803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.802931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hndz\" (UniqueName: \"kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.803254    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803337 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.803309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.803374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.803364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.805297 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.805275    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.805547 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.805531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.812013 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.811984    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hndz\" (UniqueName: \"kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz\") pod \"test-model-15d796-kserve-67d999cf7c-llb2q\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") " pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:04.990443 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:04.990350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:05.267146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.265144    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:40:05.268697 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:40:05.267744    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9355447c_cea7_48c9_8da6_aadd6c3c9e10.slice/crio-39a27f544b931a3fe420303fc96c776ce0445c0aa695b8851777100c7782e2ce WatchSource:0}: Error finding container 39a27f544b931a3fe420303fc96c776ce0445c0aa695b8851777100c7782e2ce: Status 404 returned error can't find the container with id 39a27f544b931a3fe420303fc96c776ce0445c0aa695b8851777100c7782e2ce
Sep 08 18:40:05.348517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.348487    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:05.348613 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.348545    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:05.687127 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.687092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" event={"ID":"9355447c-cea7-48c9-8da6-aadd6c3c9e10","Type":"ContainerStarted","Data":"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"}
Sep 08 18:40:05.687532 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.687139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" event={"ID":"9355447c-cea7-48c9-8da6-aadd6c3c9e10","Type":"ContainerStarted","Data":"39a27f544b931a3fe420303fc96c776ce0445c0aa695b8851777100c7782e2ce"}
Sep 08 18:40:05.704292 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:05.704241    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" podStartSLOduration=1.704226107 podStartE2EDuration="1.704226107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:05.702700128 +0000 UTC m=+980.154275264" watchObservedRunningTime="2026-09-08 18:40:05.704226107 +0000 UTC m=+980.155801238"
Sep 08 18:40:06.349326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:06.349284    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:06.349514 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:06.349350    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:07.349087 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:07.348985    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:07.349087 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:07.349057    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:08.348782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:08.348750    2582 patch_prober.go:28] interesting pod/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 08 18:40:08.348957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:08.348806    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 08 18:40:09.352833 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:09.352800    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:40:09.353282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:09.353051    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:40:09.353692 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:09.353674    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:40:14.991065 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:14.991025    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:14.991642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:14.991085    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:15.003235 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:15.003207    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:15.750440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:15.750410    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:40:42.014955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.014917    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:40:42.018887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.018864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.023504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.023480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c5ee49\"/\"kube-root-ca.crt\""
Sep 08 18:40:42.023625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.023518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c5ee49\"/\"openshift-service-ca.crt\""
Sep 08 18:40:42.023625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.023487    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c5ee49\"/\"explicit-model-c5ee49-kserve-self-signed-certs\""
Sep 08 18:40:42.023625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.023479    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c5ee49\"/\"default-dockercfg-qbwk7\""
Sep 08 18:40:42.028922 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.028622    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:40:42.046949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.046920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l284\" (UniqueName: \"kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.047086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.046965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.047086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.047015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.047086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.047035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.047086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.047067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.047265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.047111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l284\" (UniqueName: \"kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147875 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147875 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.147974 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.147914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.148123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.148102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.148210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.148132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.148270 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.148255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.150155 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.150123    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.150258 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.150137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.156486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.156468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l284\" (UniqueName: \"kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284\") pod \"explicit-model-c5ee49-kserve-659948c979-qqvxs\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") " pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.331437 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.331348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:42.603359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.603329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:40:42.605086 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:40:42.605057    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dac9ab9_04a4_4c86_9fe5_d6647f0ccfee.slice/crio-0ba638f7e283e290123e1fce314b56259bf7324d5816339a939fb158da489b5d WatchSource:0}: Error finding container 0ba638f7e283e290123e1fce314b56259bf7324d5816339a939fb158da489b5d: Status 404 returned error can't find the container with id 0ba638f7e283e290123e1fce314b56259bf7324d5816339a939fb158da489b5d
Sep 08 18:40:42.874783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.874680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" event={"ID":"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee","Type":"ContainerStarted","Data":"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"}
Sep 08 18:40:42.874783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.874736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" event={"ID":"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee","Type":"ContainerStarted","Data":"0ba638f7e283e290123e1fce314b56259bf7324d5816339a939fb158da489b5d"}
Sep 08 18:40:42.893006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:42.892956    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" podStartSLOduration=1.8929413689999999 podStartE2EDuration="1.892941369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:42.890407773 +0000 UTC m=+1017.341982901" watchObservedRunningTime="2026-09-08 18:40:42.892941369 +0000 UTC m=+1017.344516501"
Sep 08 18:40:44.800355 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:44.800311    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:40:52.331809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:52.331770    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:40:55.430966 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.430928    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:40:55.431418 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.431283    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerName="payload-processing" containerID="cri-o://2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378" gracePeriod=30
Sep 08 18:40:55.551328 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.551289    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.57:9004: connect: connection refused"
Sep 08 18:40:55.687779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.687683    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:40:55.779460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.777166    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhj8q\" (UniqueName: \"kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q\") pod \"a26d186b-93dc-4441-a2ba-f1c547aab42b\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") "
Sep 08 18:40:55.779460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.778280    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume\") pod \"a26d186b-93dc-4441-a2ba-f1c547aab42b\" (UID: \"a26d186b-93dc-4441-a2ba-f1c547aab42b\") "
Sep 08 18:40:55.779460 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.779041    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a26d186b-93dc-4441-a2ba-f1c547aab42b" (UID: "a26d186b-93dc-4441-a2ba-f1c547aab42b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:40:55.779804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.779505    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26d186b-93dc-4441-a2ba-f1c547aab42b-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.781760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.781700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q" (OuterVolumeSpecName: "kube-api-access-hhj8q") pod "a26d186b-93dc-4441-a2ba-f1c547aab42b" (UID: "a26d186b-93dc-4441-a2ba-f1c547aab42b"). InnerVolumeSpecName "kube-api-access-hhj8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:40:55.880400 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.880359    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhj8q\" (UniqueName: \"kubernetes.io/projected/a26d186b-93dc-4441-a2ba-f1c547aab42b-kube-api-access-hhj8q\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:40:55.945145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.945053    2582 generic.go:356] "Generic (PLEG): container finished" podID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerID="2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378" exitCode=0
Sep 08 18:40:55.945145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.945116    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"
Sep 08 18:40:55.945398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.945135    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" event={"ID":"a26d186b-93dc-4441-a2ba-f1c547aab42b","Type":"ContainerDied","Data":"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"}
Sep 08 18:40:55.945398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.945223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf" event={"ID":"a26d186b-93dc-4441-a2ba-f1c547aab42b","Type":"ContainerDied","Data":"2e76e5a44ba1f1e962e9869c87aa38e235053f5b6d6d9275a986dd032a47ef8d"}
Sep 08 18:40:55.945398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.945255    2582 scope.go:117] "RemoveContainer" containerID="2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"
Sep 08 18:40:55.954630 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.954614    2582 scope.go:117] "RemoveContainer" containerID="2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"
Sep 08 18:40:55.955064 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:40:55.955040    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378\": container with ID starting with 2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378 not found: ID does not exist" containerID="2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"
Sep 08 18:40:55.955161 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.955070    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378"} err="failed to get container status \"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378\": rpc error: code = NotFound desc = could not find container \"2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378\": container with ID starting with 2b4008f44b2e0695bf98465dac70bce837b0449889d0c1d2514ea2872a480378 not found: ID does not exist"
Sep 08 18:40:55.968215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.968188    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:40:55.974129 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:55.974103    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c5ee49-769f8766f-974kf"]
Sep 08 18:40:56.099904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:56.099868    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" path="/var/lib/kubelet/pods/a26d186b-93dc-4441-a2ba-f1c547aab42b/volumes"
Sep 08 18:40:59.913424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:40:59.913380    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" podUID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" containerName="main" containerID="cri-o://93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4" gracePeriod=45
Sep 08 18:41:00.168703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.168620    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:41:00.220405 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220467    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l284\" (UniqueName: \"kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220826 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache\") pod \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\" (UID: \"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee\") "
Sep 08 18:41:00.220826 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220682    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home" (OuterVolumeSpecName: "home") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:00.220826 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:00.220999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.220964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache" (OuterVolumeSpecName: "model-cache") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:00.221059 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.221003    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.221059 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.221018    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.222814 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.222791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm" (OuterVolumeSpecName: "dshm") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:00.222919 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.222869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:00.222919 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.222887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284" (OuterVolumeSpecName: "kube-api-access-6l284") pod "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" (UID: "4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee"). InnerVolumeSpecName "kube-api-access-6l284". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:00.322331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.322288    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.322331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.322324    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.322331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.322333    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l284\" (UniqueName: \"kubernetes.io/projected/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-kube-api-access-6l284\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.322331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.322341    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:00.973365 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.973330    2582 generic.go:356] "Generic (PLEG): container finished" podID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" containerID="93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4" exitCode=0
Sep 08 18:41:00.973806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.973375    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"
Sep 08 18:41:00.973806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.973417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" event={"ID":"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee","Type":"ContainerDied","Data":"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"}
Sep 08 18:41:00.973806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.973460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs" event={"ID":"4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee","Type":"ContainerDied","Data":"0ba638f7e283e290123e1fce314b56259bf7324d5816339a939fb158da489b5d"}
Sep 08 18:41:00.973806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.973477    2582 scope.go:117] "RemoveContainer" containerID="93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"
Sep 08 18:41:00.982548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.982533    2582 scope.go:117] "RemoveContainer" containerID="93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"
Sep 08 18:41:00.982830 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:00.982810    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4\": container with ID starting with 93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4 not found: ID does not exist" containerID="93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"
Sep 08 18:41:00.982892 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.982839    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4"} err="failed to get container status \"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4\": rpc error: code = NotFound desc = could not find container \"93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4\": container with ID starting with 93c2dce266fc9687f373493c1d43a2eed7e11a0c13a40bd799ff8a661f7321a4 not found: ID does not exist"
Sep 08 18:41:00.995866 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:00.995841    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:41:01.000113 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:01.000092    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c5ee49/explicit-model-c5ee49-kserve-659948c979-qqvxs"]
Sep 08 18:41:02.098702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.098663    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" path="/var/lib/kubelet/pods/4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee/volumes"
Sep 08 18:41:02.203930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.203895    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:41:02.204126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.204109    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy" containerID="cri-o://38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4" gracePeriod=30
Sep 08 18:41:02.894135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894099    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:02.894696 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894682    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" containerName="main"
Sep 08 18:41:02.894766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894700    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" containerName="main"
Sep 08 18:41:02.894766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894741    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerName="payload-processing"
Sep 08 18:41:02.894766 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894750    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerName="payload-processing"
Sep 08 18:41:02.894896 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894827    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dac9ab9-04a4-4c86-9fe5-d6647f0ccfee" containerName="main"
Sep 08 18:41:02.894896 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.894837    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26d186b-93dc-4441-a2ba-f1c547aab42b" containerName="payload-processing"
Sep 08 18:41:02.899502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.899479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:02.902403 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.902377    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-058a11-openshift-default-dockercfg-2nc2v\""
Sep 08 18:41:02.909704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:02.909678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:03.047737 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.047932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047762    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.047932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.047932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.047932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.047932 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047914    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq9rd\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.048126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.048126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.048126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.047981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq9rd\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149474    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.149987 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.149946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.150548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.150100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.150548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.150288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.152003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.151982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.152094 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.151985    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.158066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.158039    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq9rd\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.158539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.158519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token\") pod \"e2e-noten-058a11-openshift-default-c598c5b88-46x5q\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") " pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.211686 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.211651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:03.488023 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.487992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:03.488994 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:03.488966    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eb80ac_9fb2_42e3_9921_46d1dea05433.slice/crio-1d7d8ad9a44f88ac8bdf15a36b81a06083507cc1861faa5f79e817df324e4923 WatchSource:0}: Error finding container 1d7d8ad9a44f88ac8bdf15a36b81a06083507cc1861faa5f79e817df324e4923: Status 404 returned error can't find the container with id 1d7d8ad9a44f88ac8bdf15a36b81a06083507cc1861faa5f79e817df324e4923
Sep 08 18:41:03.491300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.491268    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:03.491404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.491346    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:03.491404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.491388    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:03.995232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.995194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" event={"ID":"16eb80ac-9fb2-42e3-9921-46d1dea05433","Type":"ContainerStarted","Data":"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"}
Sep 08 18:41:03.995417 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:03.995249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" event={"ID":"16eb80ac-9fb2-42e3-9921-46d1dea05433","Type":"ContainerStarted","Data":"1d7d8ad9a44f88ac8bdf15a36b81a06083507cc1861faa5f79e817df324e4923"}
Sep 08 18:41:04.021069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:04.021019    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" podStartSLOduration=2.021003328 podStartE2EDuration="2.021003328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:04.018680006 +0000 UTC m=+1038.470255150" watchObservedRunningTime="2026-09-08 18:41:04.021003328 +0000 UTC m=+1038.472578460"
Sep 08 18:41:04.212427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:04.212388    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:04.217592 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:04.217565    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:05.000302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:05.000269    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:05.001144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:05.001124    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:07.471102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.471037    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:41:07.594352 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594319    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594438    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594504    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25nmk\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594530    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594878 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594585    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594878 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert\") pod \"323c3360-4481-42d4-9e13-2aa74fa37e22\" (UID: \"323c3360-4481-42d4-9e13-2aa74fa37e22\") "
Sep 08 18:41:07.594878 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594693    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:07.594878 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594811    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data" (OuterVolumeSpecName: "istio-data") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:07.595088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.594976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:07.595088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.595045    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.595088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.595062    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.595246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.595099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:07.595415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.595387    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:07.596821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.596791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk" (OuterVolumeSpecName: "kube-api-access-25nmk") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "kube-api-access-25nmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:07.596916 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.596819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:07.597337 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.597316    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token" (OuterVolumeSpecName: "istio-token") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:07.597396 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.597324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "323c3360-4481-42d4-9e13-2aa74fa37e22" (UID: "323c3360-4481-42d4-9e13-2aa74fa37e22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:41:07.695674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695635    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323c3360-4481-42d4-9e13-2aa74fa37e22-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695666    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695675    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695683    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695965 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695691    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25nmk\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-kube-api-access-25nmk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695965 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695703    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323c3360-4481-42d4-9e13-2aa74fa37e22-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:07.695965 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:07.695711    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323c3360-4481-42d4-9e13-2aa74fa37e22-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:08.017423 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.017390    2582 generic.go:356] "Generic (PLEG): container finished" podID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerID="38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4" exitCode=0
Sep 08 18:41:08.017635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.017448    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"
Sep 08 18:41:08.017635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.017469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerDied","Data":"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"}
Sep 08 18:41:08.017635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.017554    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn" event={"ID":"323c3360-4481-42d4-9e13-2aa74fa37e22","Type":"ContainerDied","Data":"64c0ffa4db18680d5fe816d327750165177c4d79570747ee0e453221febd6a77"}
Sep 08 18:41:08.017635 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.017577    2582 scope.go:117] "RemoveContainer" containerID="38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"
Sep 08 18:41:08.027631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.027613    2582 scope.go:117] "RemoveContainer" containerID="eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"
Sep 08 18:41:08.035764 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.035707    2582 scope.go:117] "RemoveContainer" containerID="38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"
Sep 08 18:41:08.036056 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:08.036037    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4\": container with ID starting with 38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4 not found: ID does not exist" containerID="38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"
Sep 08 18:41:08.036118 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.036066    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4"} err="failed to get container status \"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4\": rpc error: code = NotFound desc = could not find container \"38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4\": container with ID starting with 38ab4cb85143034e97a2da0ad0c0d548d941676d4eee097ac5cac1b26c6864a4 not found: ID does not exist"
Sep 08 18:41:08.036118 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.036085    2582 scope.go:117] "RemoveContainer" containerID="eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"
Sep 08 18:41:08.036390 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:08.036372    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3\": container with ID starting with eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3 not found: ID does not exist" containerID="eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"
Sep 08 18:41:08.036449 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.036398    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3"} err="failed to get container status \"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3\": rpc error: code = NotFound desc = could not find container \"eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3\": container with ID starting with eef564afada703fa450f7a44790e337dc7ecd300aac9214a049d3de23ab80dd3 not found: ID does not exist"
Sep 08 18:41:08.040935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.040910    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:41:08.045293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.045268    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c5ee49-openshift-default-54899598bd-nt4wn"]
Sep 08 18:41:08.100139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:08.100106    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" path="/var/lib/kubelet/pods/323c3360-4481-42d4-9e13-2aa74fa37e22/volumes"
Sep 08 18:41:11.758004 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.757966    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:11.758385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758371    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.758443 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758388    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.758443 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758399    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.758443 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758405    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.758539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758467    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.758539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.758477    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="323c3360-4481-42d4-9e13-2aa74fa37e22" containerName="istio-proxy"
Sep 08 18:41:11.765152 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.765112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:11.768633 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.768607    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2e25b35c-dockercfg-wr4rw\""
Sep 08 18:41:11.768789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.768773    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2e25b35c\""
Sep 08 18:41:11.775275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.775251    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:11.936190 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.936150    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:11.936385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:11.936248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6p4t\" (UniqueName: \"kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.000360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.000324    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:12.003823 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.003806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.006948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.006923    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-058a11\"/\"openshift-service-ca.crt\""
Sep 08 18:41:12.006948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.006937    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-058a11\"/\"kube-root-ca.crt\""
Sep 08 18:41:12.007126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.006935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-058a11\"/\"noten-model-058a11-kserve-self-signed-certs\""
Sep 08 18:41:12.007126 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.006984    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-058a11\"/\"default-dockercfg-d8dwb\""
Sep 08 18:41:12.012195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.012173    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:12.037356 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.037322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6p4t\" (UniqueName: \"kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.037519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.037386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.038109 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.038086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.046133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.046107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6p4t\" (UniqueName: \"kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t\") pod \"payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") " pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.078139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.078095    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:12.138311 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.138446 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.138509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55k5m\" (UniqueName: \"kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.138568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.138658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.138658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.138601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k5m\" (UniqueName: \"kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239559    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.239648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.239633    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.241625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.241607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.242029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.242009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.247506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.247486    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k5m\" (UniqueName: \"kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m\") pod \"noten-model-058a11-kserve-844c88cc6c-9kmn8\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") " pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.313386 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.313286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:12.352028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.351999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:12.588389 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:12.588317    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:12.589620 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:12.589594    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97405590_bde6_40f9_8b96_1a1e1d6e6de1.slice/crio-609bad0c72a062affa3767d263786e1612e8d999b68c35ef5e1f8d55bfdb736b WatchSource:0}: Error finding container 609bad0c72a062affa3767d263786e1612e8d999b68c35ef5e1f8d55bfdb736b: Status 404 returned error can't find the container with id 609bad0c72a062affa3767d263786e1612e8d999b68c35ef5e1f8d55bfdb736b
Sep 08 18:41:13.047371 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.047335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" event={"ID":"b4ad9fb8-2faa-4359-84cd-847a3b506ec1","Type":"ContainerStarted","Data":"d12ed67425bd69217bc1486de5a25ed04f65018f0a2d9b84b0c44e014e63743f"}
Sep 08 18:41:13.047886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.047408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:13.047886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.047436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" event={"ID":"b4ad9fb8-2faa-4359-84cd-847a3b506ec1","Type":"ContainerStarted","Data":"6be5277c3179d42f9c0b5dbcd29ce84e0ad4742300b8351ef8d5575ad17cec0b"}
Sep 08 18:41:13.049113 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.049088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" event={"ID":"97405590-bde6-40f9-8b96-1a1e1d6e6de1","Type":"ContainerStarted","Data":"2a73f4541cef3acebadb280940236b76e5ad7bc6f495723e410cb86a44f9c79e"}
Sep 08 18:41:13.049243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.049118    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" event={"ID":"97405590-bde6-40f9-8b96-1a1e1d6e6de1","Type":"ContainerStarted","Data":"609bad0c72a062affa3767d263786e1612e8d999b68c35ef5e1f8d55bfdb736b"}
Sep 08 18:41:13.079035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.078985    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" podStartSLOduration=2.078968624 podStartE2EDuration="2.078968624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:13.078306302 +0000 UTC m=+1047.529881437" watchObservedRunningTime="2026-09-08 18:41:13.078968624 +0000 UTC m=+1047.530543755"
Sep 08 18:41:13.096538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:13.096484    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" podStartSLOduration=2.096466011 podStartE2EDuration="2.096466011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:13.095068074 +0000 UTC m=+1047.546643207" watchObservedRunningTime="2026-09-08 18:41:13.096466011 +0000 UTC m=+1047.548041148"
Sep 08 18:41:18.009563 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.009525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:18.115608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.115575    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:18.115825 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.115793    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerName="istio-proxy" containerID="cri-o://29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce" gracePeriod=30
Sep 08 18:41:18.796112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.796075    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:41:18.919479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.919443    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"]
Sep 08 18:41:18.923252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.923231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:18.926361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.926340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:41:18.931917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:18.931889    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"]
Sep 08 18:41:19.110148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:19.110052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x\" (UID: \"0e61a148-b4c2-4a12-8cfc-da0269e20dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:19.210980 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:19.210939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x\" (UID: \"0e61a148-b4c2-4a12-8cfc-da0269e20dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:19.211601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:19.211574    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x\" (UID: \"0e61a148-b4c2-4a12-8cfc-da0269e20dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:19.235889 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:19.235858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:19.506179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:19.506151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"]
Sep 08 18:41:19.507828 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:19.507797    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e61a148_b4c2_4a12_8cfc_da0269e20dba.slice/crio-9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde WatchSource:0}: Error finding container 9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde: Status 404 returned error can't find the container with id 9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde
Sep 08 18:41:20.002889 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:20.002846    2582 patch_prober.go:28] interesting pod/e2e-noten-058a11-openshift-default-c598c5b88-46x5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 08 18:41:20.003042 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:20.002913    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:41:20.086972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:20.086939    2582 generic.go:356] "Generic (PLEG): container finished" podID="0e61a148-b4c2-4a12-8cfc-da0269e20dba" containerID="b2453e5a9821f26b64c86b42226a702f15bf7b2bff9f03828e3ffdcfc0ef7ff8" exitCode=0
Sep 08 18:41:20.087136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:20.087022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x" event={"ID":"0e61a148-b4c2-4a12-8cfc-da0269e20dba","Type":"ContainerDied","Data":"b2453e5a9821f26b64c86b42226a702f15bf7b2bff9f03828e3ffdcfc0ef7ff8"}
Sep 08 18:41:20.087136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:20.087084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x" event={"ID":"0e61a148-b4c2-4a12-8cfc-da0269e20dba","Type":"ContainerStarted","Data":"9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde"}
Sep 08 18:41:21.228458 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:21.228432    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:21.331240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:21.331198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca\") pod \"0e61a148-b4c2-4a12-8cfc-da0269e20dba\" (UID: \"0e61a148-b4c2-4a12-8cfc-da0269e20dba\") "
Sep 08 18:41:21.331564 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:21.331542    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e61a148-b4c2-4a12-8cfc-da0269e20dba" (UID: "0e61a148-b4c2-4a12-8cfc-da0269e20dba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:21.432970 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:21.432937    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e61a148-b4c2-4a12-8cfc-da0269e20dba-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:22.098368 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:22.098335    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"
Sep 08 18:41:22.099050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:22.099020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x" event={"ID":"0e61a148-b4c2-4a12-8cfc-da0269e20dba","Type":"ContainerDied","Data":"9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde"}
Sep 08 18:41:22.099170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:22.099069    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e723409f4922eceb8f3958dcb63ceac450fdcc6021d296dbea5f1a812afedde"
Sep 08 18:41:22.313687 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:22.313650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:23.365196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.365169    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:23.449935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.449898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.449935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.449940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.449984    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450004    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450022    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450037    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq9rd\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450122    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450366    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:23.450646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450560    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data" (OuterVolumeSpecName: "istio-data") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:23.450646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:23.450646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450633    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:23.450891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450669    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token\") pod \"16eb80ac-9fb2-42e3-9921-46d1dea05433\" (UID: \"16eb80ac-9fb2-42e3-9921-46d1dea05433\") "
Sep 08 18:41:23.450891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.450783    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:23.451058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.451039    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.451114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.451065    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.451114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.451081    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.451114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.451094    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.451114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.451105    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16eb80ac-9fb2-42e3-9921-46d1dea05433-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.452747 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.452707    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token" (OuterVolumeSpecName: "istio-token") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:23.452958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.452927    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:23.452958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.452942    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd" (OuterVolumeSpecName: "kube-api-access-jq9rd") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "kube-api-access-jq9rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:23.453080 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.452963    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "16eb80ac-9fb2-42e3-9921-46d1dea05433" (UID: "16eb80ac-9fb2-42e3-9921-46d1dea05433"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:41:23.551674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.551631    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.551674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.551664    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.551674 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.551678    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq9rd\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-kube-api-access-jq9rd\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:23.551955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:23.551690    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16eb80ac-9fb2-42e3-9921-46d1dea05433-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:24.056830 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.056796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:24.110593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.110564    2582 generic.go:356] "Generic (PLEG): container finished" podID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerID="29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce" exitCode=0
Sep 08 18:41:24.110765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.110611    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"
Sep 08 18:41:24.110765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.110648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" event={"ID":"16eb80ac-9fb2-42e3-9921-46d1dea05433","Type":"ContainerDied","Data":"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"}
Sep 08 18:41:24.110765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.110748    2582 scope.go:117] "RemoveContainer" containerID="29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"
Sep 08 18:41:24.110918 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.110742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q" event={"ID":"16eb80ac-9fb2-42e3-9921-46d1dea05433","Type":"ContainerDied","Data":"1d7d8ad9a44f88ac8bdf15a36b81a06083507cc1861faa5f79e817df324e4923"}
Sep 08 18:41:24.123957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.123933    2582 scope.go:117] "RemoveContainer" containerID="29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"
Sep 08 18:41:24.124327 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:24.124299    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce\": container with ID starting with 29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce not found: ID does not exist" containerID="29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"
Sep 08 18:41:24.124409 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.124337    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce"} err="failed to get container status \"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce\": rpc error: code = NotFound desc = could not find container \"29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce\": container with ID starting with 29a20c9e1890a539d1691a6ffabdb4abea5ab03c3df52a368cfde7dd160e44ce not found: ID does not exist"
Sep 08 18:41:24.135945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.135917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:24.142137 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:24.142113    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-058a11-openshift-default-c598c5b88-46x5q"]
Sep 08 18:41:26.098998 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:26.098965    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" path="/var/lib/kubelet/pods/16eb80ac-9fb2-42e3-9921-46d1dea05433/volumes"
Sep 08 18:41:29.387588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.387555    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:41:29.388079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.387828    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" podUID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" containerName="payload-processing" containerID="cri-o://cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b" gracePeriod=30
Sep 08 18:41:29.398969 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.398940    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:41:29.399342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.399171    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" podUID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" containerName="payload-pre-processing" containerID="cri-o://3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57" gracePeriod=30
Sep 08 18:41:29.590809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.590742    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"]
Sep 08 18:41:29.593002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.592978    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1d8cb1-8mx7x"]
Sep 08 18:41:29.682872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.682844    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:41:29.700450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.700427    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:41:29.702812 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.702791    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrtx8\" (UniqueName: \"kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8\") pod \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") "
Sep 08 18:41:29.702942 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.702899    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume\") pod \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\" (UID: \"729c07d3-cb25-4322-ac62-8eccfc3bdc51\") "
Sep 08 18:41:29.703329 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.703289    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "729c07d3-cb25-4322-ac62-8eccfc3bdc51" (UID: "729c07d3-cb25-4322-ac62-8eccfc3bdc51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:29.705519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.705494    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8" (OuterVolumeSpecName: "kube-api-access-wrtx8") pod "729c07d3-cb25-4322-ac62-8eccfc3bdc51" (UID: "729c07d3-cb25-4322-ac62-8eccfc3bdc51"). InnerVolumeSpecName "kube-api-access-wrtx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:29.804336 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.804303    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzgv5\" (UniqueName: \"kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5\") pod \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") "
Sep 08 18:41:29.804507 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.804370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume\") pod \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\" (UID: \"f7d28bfa-b61d-43b1-88c2-6f34de110d8c\") "
Sep 08 18:41:29.804645 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.804627    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrtx8\" (UniqueName: \"kubernetes.io/projected/729c07d3-cb25-4322-ac62-8eccfc3bdc51-kube-api-access-wrtx8\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.804716 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.804652    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c07d3-cb25-4322-ac62-8eccfc3bdc51-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.804780 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.804712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7d28bfa-b61d-43b1-88c2-6f34de110d8c" (UID: "f7d28bfa-b61d-43b1-88c2-6f34de110d8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:29.806469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.806444    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5" (OuterVolumeSpecName: "kube-api-access-hzgv5") pod "f7d28bfa-b61d-43b1-88c2-6f34de110d8c" (UID: "f7d28bfa-b61d-43b1-88c2-6f34de110d8c"). InnerVolumeSpecName "kube-api-access-hzgv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:29.905911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.905827    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgv5\" (UniqueName: \"kubernetes.io/projected/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-kube-api-access-hzgv5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:29.905911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:29.905855    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d28bfa-b61d-43b1-88c2-6f34de110d8c-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:30.099847 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.099811    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e61a148-b4c2-4a12-8cfc-da0269e20dba" path="/var/lib/kubelet/pods/0e61a148-b4c2-4a12-8cfc-da0269e20dba/volumes"
Sep 08 18:41:30.150870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.150836    2582 generic.go:356] "Generic (PLEG): container finished" podID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" containerID="cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b" exitCode=0
Sep 08 18:41:30.151008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.150882    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"
Sep 08 18:41:30.151008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.150918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" event={"ID":"729c07d3-cb25-4322-ac62-8eccfc3bdc51","Type":"ContainerDied","Data":"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"}
Sep 08 18:41:30.151008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.150971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s" event={"ID":"729c07d3-cb25-4322-ac62-8eccfc3bdc51","Type":"ContainerDied","Data":"e2dec2b3be1cc43dc0489f1afeb7e077a2677d10ca9555a44832b16a08a6c7e9"}
Sep 08 18:41:30.151008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.151000    2582 scope.go:117] "RemoveContainer" containerID="cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"
Sep 08 18:41:30.152255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.152238    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" containerID="3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57" exitCode=0
Sep 08 18:41:30.152333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.152276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" event={"ID":"f7d28bfa-b61d-43b1-88c2-6f34de110d8c","Type":"ContainerDied","Data":"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"}
Sep 08 18:41:30.152333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.152286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"
Sep 08 18:41:30.152333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.152305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct" event={"ID":"f7d28bfa-b61d-43b1-88c2-6f34de110d8c","Type":"ContainerDied","Data":"22f91c9d541633b5cb5d57f49a7a52d5b929c7c346cfa3d5d846253236777557"}
Sep 08 18:41:30.161102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.161083    2582 scope.go:117] "RemoveContainer" containerID="cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"
Sep 08 18:41:30.161396 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:30.161376    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b\": container with ID starting with cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b not found: ID does not exist" containerID="cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"
Sep 08 18:41:30.161466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.161404    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b"} err="failed to get container status \"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b\": rpc error: code = NotFound desc = could not find container \"cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b\": container with ID starting with cda92bca55861a70b5c4fe8a8f845adb3ef5a2a321f346a1a783917dd248b04b not found: ID does not exist"
Sep 08 18:41:30.161466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.161427    2582 scope.go:117] "RemoveContainer" containerID="3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"
Sep 08 18:41:30.169243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.169227    2582 scope.go:117] "RemoveContainer" containerID="3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"
Sep 08 18:41:30.169514 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:30.169498    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57\": container with ID starting with 3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57 not found: ID does not exist" containerID="3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"
Sep 08 18:41:30.169562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.169521    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57"} err="failed to get container status \"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57\": rpc error: code = NotFound desc = could not find container \"3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57\": container with ID starting with 3427d54b6ac13cd6fae97d004fafca7a788bf7169df1d9f13211d5aa7aaeef57 not found: ID does not exist"
Sep 08 18:41:30.199086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.199053    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:41:30.214749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.211044    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1d8cb1-5d67cbfb6-6fxct"]
Sep 08 18:41:30.239771 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.239717    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:41:30.244960 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:30.244934    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1d8cb1-568bf6b67f-5ht8s"]
Sep 08 18:41:32.099547 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:32.099509    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" path="/var/lib/kubelet/pods/729c07d3-cb25-4322-ac62-8eccfc3bdc51/volumes"
Sep 08 18:41:32.099954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:32.099878    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" path="/var/lib/kubelet/pods/f7d28bfa-b61d-43b1-88c2-6f34de110d8c/volumes"
Sep 08 18:41:33.030920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.030881    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" podUID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" containerName="main" containerID="cri-o://2a73f4541cef3acebadb280940236b76e5ad7bc6f495723e410cb86a44f9c79e" gracePeriod=45
Sep 08 18:41:33.172242 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.172202    2582 generic.go:356] "Generic (PLEG): container finished" podID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" containerID="2a73f4541cef3acebadb280940236b76e5ad7bc6f495723e410cb86a44f9c79e" exitCode=0
Sep 08 18:41:33.172614 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.172281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" event={"ID":"97405590-bde6-40f9-8b96-1a1e1d6e6de1","Type":"ContainerDied","Data":"2a73f4541cef3acebadb280940236b76e5ad7bc6f495723e410cb86a44f9c79e"}
Sep 08 18:41:33.283199 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.283133    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:33.337426 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.337612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.337612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337480    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.337612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337670    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55k5m\" (UniqueName: \"kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m\") pod \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\" (UID: \"97405590-bde6-40f9-8b96-1a1e1d6e6de1\") "
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337658    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache" (OuterVolumeSpecName: "model-cache") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337901    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home" (OuterVolumeSpecName: "home") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337964    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.338038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.337983    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.339955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.339929    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:33.339955 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.339933    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm" (OuterVolumeSpecName: "dshm") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:33.340148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.340133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m" (OuterVolumeSpecName: "kube-api-access-55k5m") pod "97405590-bde6-40f9-8b96-1a1e1d6e6de1" (UID: "97405590-bde6-40f9-8b96-1a1e1d6e6de1"). InnerVolumeSpecName "kube-api-access-55k5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:33.438608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.438570    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.438608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.438600    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55k5m\" (UniqueName: \"kubernetes.io/projected/97405590-bde6-40f9-8b96-1a1e1d6e6de1-kube-api-access-55k5m\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.438608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.438612    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97405590-bde6-40f9-8b96-1a1e1d6e6de1-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.438608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.438620    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97405590-bde6-40f9-8b96-1a1e1d6e6de1-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:33.817142 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:33.817099    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" podUID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" containerName="main" containerID="cri-o://0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29" gracePeriod=45
Sep 08 18:41:34.078195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.078124    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:41:34.146591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146554    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.146803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.146803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.146803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146751    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hndz\" (UniqueName: \"kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.146803 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146791    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.147043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146824    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir\") pod \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\" (UID: \"9355447c-cea7-48c9-8da6-aadd6c3c9e10\") "
Sep 08 18:41:34.147043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache" (OuterVolumeSpecName: "model-cache") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:34.147043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.146902    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home" (OuterVolumeSpecName: "home") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:34.147186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.147109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:34.147186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.147114    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:34.147186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.147146    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:34.149116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.149091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm" (OuterVolumeSpecName: "dshm") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:34.149481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.149462    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:41:34.149481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.149466    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz" (OuterVolumeSpecName: "kube-api-access-6hndz") pod "9355447c-cea7-48c9-8da6-aadd6c3c9e10" (UID: "9355447c-cea7-48c9-8da6-aadd6c3c9e10"). InnerVolumeSpecName "kube-api-access-6hndz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:34.178800 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.178758    2582 generic.go:356] "Generic (PLEG): container finished" podID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" containerID="0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29" exitCode=0
Sep 08 18:41:34.179166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.178805    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" event={"ID":"9355447c-cea7-48c9-8da6-aadd6c3c9e10","Type":"ContainerDied","Data":"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"}
Sep 08 18:41:34.179166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.178831    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"
Sep 08 18:41:34.179166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.178852    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q" event={"ID":"9355447c-cea7-48c9-8da6-aadd6c3c9e10","Type":"ContainerDied","Data":"39a27f544b931a3fe420303fc96c776ce0445c0aa695b8851777100c7782e2ce"}
Sep 08 18:41:34.179166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.178874    2582 scope.go:117] "RemoveContainer" containerID="0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"
Sep 08 18:41:34.180661 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.180638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8" event={"ID":"97405590-bde6-40f9-8b96-1a1e1d6e6de1","Type":"ContainerDied","Data":"609bad0c72a062affa3767d263786e1612e8d999b68c35ef5e1f8d55bfdb736b"}
Sep 08 18:41:34.180661 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.180655    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"
Sep 08 18:41:34.188819 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.188794    2582 scope.go:117] "RemoveContainer" containerID="0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"
Sep 08 18:41:34.189105 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:34.189081    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29\": container with ID starting with 0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29 not found: ID does not exist" containerID="0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"
Sep 08 18:41:34.189217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.189117    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29"} err="failed to get container status \"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29\": rpc error: code = NotFound desc = could not find container \"0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29\": container with ID starting with 0cf2dcb6e1e3b51beb7d730ce4a114ba2901c775a7d2276b3b1205f1f7ab1d29 not found: ID does not exist"
Sep 08 18:41:34.189217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.189140    2582 scope.go:117] "RemoveContainer" containerID="2a73f4541cef3acebadb280940236b76e5ad7bc6f495723e410cb86a44f9c79e"
Sep 08 18:41:34.198829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.198809    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:34.202343 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.202322    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-058a11/noten-model-058a11-kserve-844c88cc6c-9kmn8"]
Sep 08 18:41:34.214381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.214121    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:41:34.216072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.216054    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-15d796/test-model-15d796-kserve-67d999cf7c-llb2q"]
Sep 08 18:41:34.247934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.247893    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:34.247934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.247926    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hndz\" (UniqueName: \"kubernetes.io/projected/9355447c-cea7-48c9-8da6-aadd6c3c9e10-kube-api-access-6hndz\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:34.247934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.247936    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:34.248127 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:34.247948    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9355447c-cea7-48c9-8da6-aadd6c3c9e10-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:36.098746 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:36.098682    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" path="/var/lib/kubelet/pods/9355447c-cea7-48c9-8da6-aadd6c3c9e10/volumes"
Sep 08 18:41:36.099262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:36.099238    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" path="/var/lib/kubelet/pods/97405590-bde6-40f9-8b96-1a1e1d6e6de1/volumes"
Sep 08 18:41:36.635764 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:36.635707    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:41:36.635974 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:36.635951    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy" containerID="cri-o://e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70" gracePeriod=30
Sep 08 18:41:37.005364 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005318    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"]
Sep 08 18:41:37.005759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005741    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" containerName="payload-pre-processing"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005762    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" containerName="payload-pre-processing"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005785    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" containerName="payload-processing"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005794    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" containerName="payload-processing"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005806    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" containerName="main"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005814    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" containerName="main"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005826    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerName="istio-proxy"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005834    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerName="istio-proxy"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005847    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" containerName="main"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" containerName="main"
Sep 08 18:41:37.005865 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005863    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e61a148-b4c2-4a12-8cfc-da0269e20dba" containerName="revoke-keys"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005871    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e61a148-b4c2-4a12-8cfc-da0269e20dba" containerName="revoke-keys"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005960    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="729c07d3-cb25-4322-ac62-8eccfc3bdc51" containerName="payload-processing"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005973    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9355447c-cea7-48c9-8da6-aadd6c3c9e10" containerName="main"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005985    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="16eb80ac-9fb2-42e3-9921-46d1dea05433" containerName="istio-proxy"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.005995    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e61a148-b4c2-4a12-8cfc-da0269e20dba" containerName="revoke-keys"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.006003    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="97405590-bde6-40f9-8b96-1a1e1d6e6de1" containerName="main"
Sep 08 18:41:37.006339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.006011    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d28bfa-b61d-43b1-88c2-6f34de110d8c" containerName="payload-pre-processing"
Sep 08 18:41:37.010555 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.010532    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:37.013835 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.013812    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:41:37.016575 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.016463    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"]
Sep 08 18:41:37.077661 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.077618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx\" (UID: \"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:37.178664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.178619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx\" (UID: \"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:37.179390 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.179361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx\" (UID: \"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:37.322399 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.322306    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:37.592928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:37.592904    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"]
Sep 08 18:41:37.595417 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:37.595381    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e9f015_acfd_4e6e_ab11_773cfcb39b5a.slice/crio-78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7 WatchSource:0}: Error finding container 78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7: Status 404 returned error can't find the container with id 78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7
Sep 08 18:41:38.204318 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:38.204283    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx" event={"ID":"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a","Type":"ContainerStarted","Data":"df39ac62477785f79d423911b635877a084479d2ec1e73ca3c6bbcbf1d45ab01"}
Sep 08 18:41:38.204705 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:38.204339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx" event={"ID":"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a","Type":"ContainerStarted","Data":"78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7"}
Sep 08 18:41:38.222419 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:38.222368    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx" podStartSLOduration=2.222353129 podStartE2EDuration="2.222353129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:38.221598764 +0000 UTC m=+1072.673173895" watchObservedRunningTime="2026-09-08 18:41:38.222353129 +0000 UTC m=+1072.673928292"
Sep 08 18:41:39.210308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:39.210273    2582 generic.go:356] "Generic (PLEG): container finished" podID="f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" containerID="df39ac62477785f79d423911b635877a084479d2ec1e73ca3c6bbcbf1d45ab01" exitCode=0
Sep 08 18:41:39.210853 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:39.210372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx" event={"ID":"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a","Type":"ContainerDied","Data":"df39ac62477785f79d423911b635877a084479d2ec1e73ca3c6bbcbf1d45ab01"}
Sep 08 18:41:40.345776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:40.345752    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:40.405284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:40.405254    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca\") pod \"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a\" (UID: \"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a\") "
Sep 08 18:41:40.405624 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:40.405599    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" (UID: "f3e9f015-acfd-4e6e-ab11-773cfcb39b5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:40.506247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:40.506159    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:41.221670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.221635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"
Sep 08 18:41:41.221857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.221641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx" event={"ID":"f3e9f015-acfd-4e6e-ab11-773cfcb39b5a","Type":"ContainerDied","Data":"78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7"}
Sep 08 18:41:41.221857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.221782    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78970b4b4661c6bf4aabbc1a46ee9c097707a33780d24ee2ae1c8ceedd0360a7"
Sep 08 18:41:41.885617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.885595    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:41:41.919717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.919717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.919717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.919717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919822    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919899    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s82n\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy\") pod \"80dba3e7-e068-491a-beb1-315f1e29a7b5\" (UID: \"80dba3e7-e068-491a-beb1-315f1e29a7b5\") "
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.919991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920030    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:41.920153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920145    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:41.920599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920333    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:41.920599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:41.920599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920358    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:41.920599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920410    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:41.920599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.920433    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:41.922265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.922235    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:41.922481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.922463    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:41:41.922843 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.922821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:41:41.922943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:41.922850    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n" (OuterVolumeSpecName: "kube-api-access-5s82n") pod "80dba3e7-e068-491a-beb1-315f1e29a7b5" (UID: "80dba3e7-e068-491a-beb1-315f1e29a7b5"). InnerVolumeSpecName "kube-api-access-5s82n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:42.021041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.021003    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:42.021041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.021033    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:42.021041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.021043    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80dba3e7-e068-491a-beb1-315f1e29a7b5-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:42.021273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.021055    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s82n\" (UniqueName: \"kubernetes.io/projected/80dba3e7-e068-491a-beb1-315f1e29a7b5-kube-api-access-5s82n\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:42.021273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.021064    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80dba3e7-e068-491a-beb1-315f1e29a7b5-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:42.228642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.228551    2582 generic.go:356] "Generic (PLEG): container finished" podID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerID="e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70" exitCode=0
Sep 08 18:41:42.228642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.228585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerDied","Data":"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"}
Sep 08 18:41:42.228642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.228619    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"
Sep 08 18:41:42.228913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.228643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm" event={"ID":"80dba3e7-e068-491a-beb1-315f1e29a7b5","Type":"ContainerDied","Data":"a62edc928baf7275f0244a7387d932bf1c1fd39d668660af30f748d1f9aa6769"}
Sep 08 18:41:42.228913 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.228678    2582 scope.go:117] "RemoveContainer" containerID="e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"
Sep 08 18:41:42.237788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.237769    2582 scope.go:117] "RemoveContainer" containerID="039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"
Sep 08 18:41:42.245812 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.245794    2582 scope.go:117] "RemoveContainer" containerID="e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"
Sep 08 18:41:42.246072 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:42.246053    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70\": container with ID starting with e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70 not found: ID does not exist" containerID="e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"
Sep 08 18:41:42.246132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.246081    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70"} err="failed to get container status \"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70\": rpc error: code = NotFound desc = could not find container \"e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70\": container with ID starting with e080202612f7dd85d15ecf5fa91280865b4fc7485544fd21403251a466324a70 not found: ID does not exist"
Sep 08 18:41:42.246132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.246097    2582 scope.go:117] "RemoveContainer" containerID="039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"
Sep 08 18:41:42.246333 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:41:42.246316    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b\": container with ID starting with 039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b not found: ID does not exist" containerID="039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"
Sep 08 18:41:42.246374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.246338    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b"} err="failed to get container status \"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b\": rpc error: code = NotFound desc = could not find container \"039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b\": container with ID starting with 039b05862c6d4774ee30487b27ea48d8dd8991eaa935a2089997563e0e41520b not found: ID does not exist"
Sep 08 18:41:42.268357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.268330    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:41:42.285344 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.285315    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1d8cb1-openshift-default-7c489cdd44-8ddrm"]
Sep 08 18:41:42.412682 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.412638    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"]
Sep 08 18:41:42.413099 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413080    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:42.413183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413101    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:42.413183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413113    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" containerName="revoke-keys"
Sep 08 18:41:42.413183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413121    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" containerName="revoke-keys"
Sep 08 18:41:42.413334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413195    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:42.413334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.413211    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" containerName="revoke-keys"
Sep 08 18:41:42.417451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.417429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:42.420790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.420763    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:41:42.424015 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.423950    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"]
Sep 08 18:41:42.524529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.524422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn\" (UID: \"dc3a03df-7105-4a51-8b03-a885af9395c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:42.625528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.625412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn\" (UID: \"dc3a03df-7105-4a51-8b03-a885af9395c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:42.626086 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.626062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn\" (UID: \"dc3a03df-7105-4a51-8b03-a885af9395c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:42.730620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:42.730585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:43.006852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:43.006707    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"]
Sep 08 18:41:43.009761 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:43.009705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3a03df_7105_4a51_8b03_a885af9395c6.slice/crio-34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95 WatchSource:0}: Error finding container 34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95: Status 404 returned error can't find the container with id 34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95
Sep 08 18:41:43.235183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:43.235146    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn" event={"ID":"dc3a03df-7105-4a51-8b03-a885af9395c6","Type":"ContainerStarted","Data":"34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95"}
Sep 08 18:41:44.099827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:44.099710    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" path="/var/lib/kubelet/pods/80dba3e7-e068-491a-beb1-315f1e29a7b5/volumes"
Sep 08 18:41:44.242660 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:44.242623    2582 generic.go:356] "Generic (PLEG): container finished" podID="dc3a03df-7105-4a51-8b03-a885af9395c6" containerID="c5f3ba3343879670537e18ba6c32722b5089fbfc1a2f390f3e0e1d70ae57904e" exitCode=0
Sep 08 18:41:44.242846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:44.242707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn" event={"ID":"dc3a03df-7105-4a51-8b03-a885af9395c6","Type":"ContainerDied","Data":"c5f3ba3343879670537e18ba6c32722b5089fbfc1a2f390f3e0e1d70ae57904e"}
Sep 08 18:41:45.379222 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:45.379192    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:45.449848 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:45.449820    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca\") pod \"dc3a03df-7105-4a51-8b03-a885af9395c6\" (UID: \"dc3a03df-7105-4a51-8b03-a885af9395c6\") "
Sep 08 18:41:45.450175 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:45.450149    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc3a03df-7105-4a51-8b03-a885af9395c6" (UID: "dc3a03df-7105-4a51-8b03-a885af9395c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:45.550943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:45.550850    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc3a03df-7105-4a51-8b03-a885af9395c6-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:46.253456 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:46.253429    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"
Sep 08 18:41:46.253598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:46.253428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn" event={"ID":"dc3a03df-7105-4a51-8b03-a885af9395c6","Type":"ContainerDied","Data":"34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95"}
Sep 08 18:41:46.253598 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:46.253560    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34fc0d5c5d5f77a06dd5ebe7610257dc16acc531a29884584e196a1ad6039f95"
Sep 08 18:41:47.967874 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:47.967840    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"]
Sep 08 18:41:47.972569 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:47.972536    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15d796-mnvlx"]
Sep 08 18:41:48.100001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:48.099966    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3e9f015-acfd-4e6e-ab11-773cfcb39b5a" path="/var/lib/kubelet/pods/f3e9f015-acfd-4e6e-ab11-773cfcb39b5a/volumes"
Sep 08 18:41:53.125768 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.125709    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:53.126160 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.125987    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" podUID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" containerName="payload-processing" containerID="cri-o://d12ed67425bd69217bc1486de5a25ed04f65018f0a2d9b84b0c44e014e63743f" gracePeriod=30
Sep 08 18:41:53.276392 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.276354    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"]
Sep 08 18:41:53.277941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.277906    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2e25b35c-d5ncn"]
Sep 08 18:41:53.291265 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.291231    2582 generic.go:356] "Generic (PLEG): container finished" podID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" containerID="d12ed67425bd69217bc1486de5a25ed04f65018f0a2d9b84b0c44e014e63743f" exitCode=0
Sep 08 18:41:53.291415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.291315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" event={"ID":"b4ad9fb8-2faa-4359-84cd-847a3b506ec1","Type":"ContainerDied","Data":"d12ed67425bd69217bc1486de5a25ed04f65018f0a2d9b84b0c44e014e63743f"}
Sep 08 18:41:53.383872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.383798    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:53.521207 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.521168    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6p4t\" (UniqueName: \"kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t\") pod \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") "
Sep 08 18:41:53.521207 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.521212    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume\") pod \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\" (UID: \"b4ad9fb8-2faa-4359-84cd-847a3b506ec1\") "
Sep 08 18:41:53.521630 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.521607    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4ad9fb8-2faa-4359-84cd-847a3b506ec1" (UID: "b4ad9fb8-2faa-4359-84cd-847a3b506ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:41:53.523424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.523401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t" (OuterVolumeSpecName: "kube-api-access-v6p4t") pod "b4ad9fb8-2faa-4359-84cd-847a3b506ec1" (UID: "b4ad9fb8-2faa-4359-84cd-847a3b506ec1"). InnerVolumeSpecName "kube-api-access-v6p4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:41:53.622053 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.622013    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6p4t\" (UniqueName: \"kubernetes.io/projected/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-kube-api-access-v6p4t\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:53.622053 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:53.622045    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ad9fb8-2faa-4359-84cd-847a3b506ec1-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:41:54.098971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.098935    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3a03df-7105-4a51-8b03-a885af9395c6" path="/var/lib/kubelet/pods/dc3a03df-7105-4a51-8b03-a885af9395c6/volumes"
Sep 08 18:41:54.297719 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.297682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz" event={"ID":"b4ad9fb8-2faa-4359-84cd-847a3b506ec1","Type":"ContainerDied","Data":"6be5277c3179d42f9c0b5dbcd29ce84e0ad4742300b8351ef8d5575ad17cec0b"}
Sep 08 18:41:54.297719 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.297708    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"
Sep 08 18:41:54.298271 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.297772    2582 scope.go:117] "RemoveContainer" containerID="d12ed67425bd69217bc1486de5a25ed04f65018f0a2d9b84b0c44e014e63743f"
Sep 08 18:41:54.315760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.315714    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:54.322992 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:54.322967    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2e25b35c-778dc4c9b5-j75zz"]
Sep 08 18:41:55.560360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560324    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:41:55.560935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560883    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3a03df-7105-4a51-8b03-a885af9395c6" containerName="revoke-keys"
Sep 08 18:41:55.560935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560904    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3a03df-7105-4a51-8b03-a885af9395c6" containerName="revoke-keys"
Sep 08 18:41:55.560935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560931    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" containerName="payload-processing"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560940    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" containerName="payload-processing"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560961    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.560969    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.561054    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="80dba3e7-e068-491a-beb1-315f1e29a7b5" containerName="istio-proxy"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.561066    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3a03df-7105-4a51-8b03-a885af9395c6" containerName="revoke-keys"
Sep 08 18:41:55.561100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.561077    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" containerName="payload-processing"
Sep 08 18:41:55.565386 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.565368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.568544 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.568521    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-792c35-openshift-default-dockercfg-6g96z\""
Sep 08 18:41:55.575129 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.575106    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:41:55.639261 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639270    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639543 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639543 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6snx\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639543 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639497    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639543 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.639667 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.639597    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740097 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6snx\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740533 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740533 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740465    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740533 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.740789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.740776    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.741034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.741016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.741105 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.741016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.741213 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.741194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.743142 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.743118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.743366 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.743347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.749349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.749318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6snx\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.750089 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.750068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token\") pod \"e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") " pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:55.877859 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:55.877767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:56.099561 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.099531    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ad9fb8-2faa-4359-84cd-847a3b506ec1" path="/var/lib/kubelet/pods/b4ad9fb8-2faa-4359-84cd-847a3b506ec1/volumes"
Sep 08 18:41:56.182116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.182086    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:41:56.182476 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:41:56.182450    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b5272a_bae3_446a_a66a_b5e80fef86d8.slice/crio-c44a06eb0b6904f1b2ee7a99e457f220e3989223d3f7fbe63179927c8b58e751 WatchSource:0}: Error finding container c44a06eb0b6904f1b2ee7a99e457f220e3989223d3f7fbe63179927c8b58e751: Status 404 returned error can't find the container with id c44a06eb0b6904f1b2ee7a99e457f220e3989223d3f7fbe63179927c8b58e751
Sep 08 18:41:56.184654 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.184616    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:56.184789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.184698    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:56.184789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.184748    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:41:56.309349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.309315    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerStarted","Data":"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"}
Sep 08 18:41:56.309546 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.309372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerStarted","Data":"c44a06eb0b6904f1b2ee7a99e457f220e3989223d3f7fbe63179927c8b58e751"}
Sep 08 18:41:56.335388 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.335338    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podStartSLOduration=1.335321445 podStartE2EDuration="1.335321445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:56.333607362 +0000 UTC m=+1090.785182495" watchObservedRunningTime="2026-09-08 18:41:56.335321445 +0000 UTC m=+1090.786896577"
Sep 08 18:41:56.878661 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:56.878621    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:57.883899 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:57.883861    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:58.320797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:58.320758    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:41:58.321705 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:41:58.321682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:42:00.387092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.387055    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:42:00.390467 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.390449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.393953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.393930    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-792c35-dockercfg-wv6fr\""
Sep 08 18:42:00.394904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.394882    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-792c35\""
Sep 08 18:42:00.408417 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.408375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:42:00.410130 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.410103    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:42:00.413393 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.413368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.426643 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.426618    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:42:00.484260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.484226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.484260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.484265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpwc\" (UniqueName: \"kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.585114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.585074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.585114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.585119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpwc\" (UniqueName: \"kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.585352 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.585252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmrn\" (UniqueName: \"kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.585417 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.585397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.585810 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.585795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.597251 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.597223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpwc\" (UniqueName: \"kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc\") pod \"payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.686628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.686517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.686811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.686631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmrn\" (UniqueName: \"kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.687340 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.687318    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.696297 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.696267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmrn\" (UniqueName: \"kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn\") pod \"payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.702116 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.702094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:00.727357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.727260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:00.984999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:00.984970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:42:00.987189 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:42:00.987162    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc50723_0425_4388_9781_9b156156da5c.slice/crio-2abb9381d4f59d2fe5866b4b8af2718e6df0e72a1bb099c266f419d42c865782 WatchSource:0}: Error finding container 2abb9381d4f59d2fe5866b4b8af2718e6df0e72a1bb099c266f419d42c865782: Status 404 returned error can't find the container with id 2abb9381d4f59d2fe5866b4b8af2718e6df0e72a1bb099c266f419d42c865782
Sep 08 18:42:01.017935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.017910    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:42:01.019755 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:42:01.019704    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb36e503_d790_4cca_916f_12fd0d5d6e8c.slice/crio-fcc53867d7e854eac55981791f657514c9febec1b95c8704a752996bd07bf370 WatchSource:0}: Error finding container fcc53867d7e854eac55981791f657514c9febec1b95c8704a752996bd07bf370: Status 404 returned error can't find the container with id fcc53867d7e854eac55981791f657514c9febec1b95c8704a752996bd07bf370
Sep 08 18:42:01.337564 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.337464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" event={"ID":"3fc50723-0425-4388-9781-9b156156da5c","Type":"ContainerStarted","Data":"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"}
Sep 08 18:42:01.337564 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.337531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" event={"ID":"3fc50723-0425-4388-9781-9b156156da5c","Type":"ContainerStarted","Data":"2abb9381d4f59d2fe5866b4b8af2718e6df0e72a1bb099c266f419d42c865782"}
Sep 08 18:42:01.337836 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.337650    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:01.339246 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.339225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" event={"ID":"eb36e503-d790-4cca-916f-12fd0d5d6e8c","Type":"ContainerStarted","Data":"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"}
Sep 08 18:42:01.339349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.339261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" event={"ID":"eb36e503-d790-4cca-916f-12fd0d5d6e8c","Type":"ContainerStarted","Data":"fcc53867d7e854eac55981791f657514c9febec1b95c8704a752996bd07bf370"}
Sep 08 18:42:01.339389 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.339357    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:01.360232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.360166    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" podStartSLOduration=1.360145844 podStartE2EDuration="1.360145844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:01.356865056 +0000 UTC m=+1095.808440188" watchObservedRunningTime="2026-09-08 18:42:01.360145844 +0000 UTC m=+1095.811720981"
Sep 08 18:42:01.377042 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:01.376992    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" podStartSLOduration=1.376976343 podStartE2EDuration="1.376976343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:01.375998749 +0000 UTC m=+1095.827573894" watchObservedRunningTime="2026-09-08 18:42:01.376976343 +0000 UTC m=+1095.828551475"
Sep 08 18:42:12.346215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:12.346179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:42:12.346628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:12.346418    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:42:13.322581 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.322491    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:13.322581 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.322553    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:13.412789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.412758    2582 generic.go:356] "Generic (PLEG): container finished" podID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerID="b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97" exitCode=0
Sep 08 18:42:13.413179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.412839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerDied","Data":"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"}
Sep 08 18:42:13.413243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.413229    2582 scope.go:117] "RemoveContainer" containerID="b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"
Sep 08 18:42:13.414301 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.414271    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:13.414362 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.414348    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:13.414400 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:13.414384    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:14.329271 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.329235    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:42:14.332821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.332801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.335531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.335507    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-3b04d6-openshift-default-dockercfg-dhpt6\""
Sep 08 18:42:14.349715 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.349679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:42:14.423036 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.423004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerStarted","Data":"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"}
Sep 08 18:42:14.514326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514396    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.514685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.514587    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfzh\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.615906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.615906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.615906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.615906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.615995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfzh\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616535 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616744 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.616953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.616932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.618816 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.618797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.618945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.618924    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.625166 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.625145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.625308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.625286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfzh\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh\") pod \"e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") " pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.645173 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.645139    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:14.879811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.879110    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:42:14.879811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.879234    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:14.879811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.879288    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:14.950072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.950043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:42:14.951871 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:42:14.951846    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0475295f_9232_4171_ae9a_8965992177da.slice/crio-b11d094eb28fef4f0794485a8cbbef0594ebac4f8b6a0a2842c07698eb96b811 WatchSource:0}: Error finding container b11d094eb28fef4f0794485a8cbbef0594ebac4f8b6a0a2842c07698eb96b811: Status 404 returned error can't find the container with id b11d094eb28fef4f0794485a8cbbef0594ebac4f8b6a0a2842c07698eb96b811
Sep 08 18:42:14.954192 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.954162    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:14.954283 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.954224    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:14.954283 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:14.954251    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:15.431006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.430498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerStarted","Data":"caa94830df8953fa5017e08bcf5db0ea1bb25c5d9a8ee09f4b0b7621d6dfec9e"}
Sep 08 18:42:15.431006 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.430557    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerStarted","Data":"b11d094eb28fef4f0794485a8cbbef0594ebac4f8b6a0a2842c07698eb96b811"}
Sep 08 18:42:15.453404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.453349    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podStartSLOduration=1.453332973 podStartE2EDuration="1.453332973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:15.452137446 +0000 UTC m=+1109.903712595" watchObservedRunningTime="2026-09-08 18:42:15.453332973 +0000 UTC m=+1109.904908105"
Sep 08 18:42:15.645900 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.645854    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:15.879158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.878583    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:15.879158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:15.878653    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:16.651566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:16.651535    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:16.878387 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:16.878347    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:16.878670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:16.878638    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:17.440738 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:17.440698    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:17.441656 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:17.441639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:17.878604 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:17.878499    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:17.878604 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:17.878582    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:18.879256 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:18.879214    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 08 18:42:18.879776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:18.879299    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 08 18:42:19.882471 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:19.882435    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:42:19.882965 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:19.882818    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:42:19.883413 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:19.883394    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:42:29.166749 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.165930    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:42:29.171103 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.171077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.185049 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.185013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6255f2-dockercfg-ckcmm\""
Sep 08 18:42:29.185615 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.185584    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:42:29.186935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.186662    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6255f2\""
Sep 08 18:42:29.249159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.249106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6bq\" (UniqueName: \"kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.249361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.249230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.350544 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.350504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.350752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.350591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6bq\" (UniqueName: \"kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.351236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.351215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.360703 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.360653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6bq\" (UniqueName: \"kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq\") pod \"payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.495994 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.495911    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:29.775349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:29.775316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:42:29.777658 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:42:29.777623    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b216c7_f6f5_4f15_bd2f_9222bb5f167a.slice/crio-fd9162a8c503e8d06f4ee52b1687940c11d23a923469801df2bc015e25cc70c8 WatchSource:0}: Error finding container fd9162a8c503e8d06f4ee52b1687940c11d23a923469801df2bc015e25cc70c8: Status 404 returned error can't find the container with id fd9162a8c503e8d06f4ee52b1687940c11d23a923469801df2bc015e25cc70c8
Sep 08 18:42:30.513967 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:30.513933    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" event={"ID":"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a","Type":"ContainerStarted","Data":"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"}
Sep 08 18:42:30.514478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:30.513991    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:30.514478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:30.514009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" event={"ID":"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a","Type":"ContainerStarted","Data":"fd9162a8c503e8d06f4ee52b1687940c11d23a923469801df2bc015e25cc70c8"}
Sep 08 18:42:30.533410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:30.533350    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" podStartSLOduration=1.5333302309999999 podStartE2EDuration="1.533330231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:30.532894007 +0000 UTC m=+1124.984469133" watchObservedRunningTime="2026-09-08 18:42:30.533330231 +0000 UTC m=+1124.984905363"
Sep 08 18:42:32.676087 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:32.676001    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": read tcp 10.134.0.2:42930->10.134.0.69:15021: read: connection reset by peer" start-of-body=
Sep 08 18:42:32.676479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:32.676088    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": read tcp 10.134.0.2:42930->10.134.0.69:15021: read: connection reset by peer"
Sep 08 18:42:33.531248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.531213    2582 generic.go:356] "Generic (PLEG): container finished" podID="0475295f-9232-4171-ae9a-8965992177da" containerID="caa94830df8953fa5017e08bcf5db0ea1bb25c5d9a8ee09f4b0b7621d6dfec9e" exitCode=0
Sep 08 18:42:33.531248 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.531251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerDied","Data":"caa94830df8953fa5017e08bcf5db0ea1bb25c5d9a8ee09f4b0b7621d6dfec9e"}
Sep 08 18:42:33.531649 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.531636    2582 scope.go:117] "RemoveContainer" containerID="caa94830df8953fa5017e08bcf5db0ea1bb25c5d9a8ee09f4b0b7621d6dfec9e"
Sep 08 18:42:33.532711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.532683    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:33.532841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.532789    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:33.532909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:33.532840    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:34.538053 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:34.538018    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerStarted","Data":"0db9d1775b92f2494ad9106c2ad02f97715ae3283fd1b6b44f67c3fadfc94a16"}
Sep 08 18:42:34.645903 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:34.645865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:34.646133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:34.646111    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:34.646184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:34.646158    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:35.646267 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:35.646223    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:35.646809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:35.646290    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:36.645894 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:36.645855    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:36.646071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:36.645933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:37.645584 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:37.645543    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:37.646017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:37.645618    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:38.646373 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:38.646339    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:38.646781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:38.646413    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:39.645621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:39.645584    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 08 18:42:39.645841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:39.645651    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 08 18:42:40.649552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:40.649515    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:40.649996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:40.649763    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:40.650416 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:40.650399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:42:41.521943 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:41.521911    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:42:42.891958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.891924    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:42:42.894482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.894460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.897468 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.897441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-951c3e-openshift-default-dockercfg-xnldz\""
Sep 08 18:42:42.907929 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.907899    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:42:42.979489 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.979702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.979702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.979702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.979702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.980026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.980026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.980026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:42.980026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:42.979932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmkv\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.080911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.080871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmkv\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.080918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.080962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.080982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081422    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.081855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.081836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.083764 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.083707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.083895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.083878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.090983 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.090955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.091469 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.091450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmkv\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv\") pod \"e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") " pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.205361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.205260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:43.481785 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.481755    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:42:43.483114 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:42:43.483089    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb75f18a_5e46_4c5b_a9e7_0e179a3ea0fe.slice/crio-bcc523c17d065f95b7ca1bc1b3babc53979c225cef017fede4d89a34d9192724 WatchSource:0}: Error finding container bcc523c17d065f95b7ca1bc1b3babc53979c225cef017fede4d89a34d9192724: Status 404 returned error can't find the container with id bcc523c17d065f95b7ca1bc1b3babc53979c225cef017fede4d89a34d9192724
Sep 08 18:42:43.485415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.485384    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:43.485487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.485453    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:43.485528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.485489    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:42:43.595026 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.594989    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerStarted","Data":"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"}
Sep 08 18:42:43.595229 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.595053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerStarted","Data":"bcc523c17d065f95b7ca1bc1b3babc53979c225cef017fede4d89a34d9192724"}
Sep 08 18:42:43.617548 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:43.617492    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podStartSLOduration=1.617473381 podStartE2EDuration="1.617473381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:42:43.615986993 +0000 UTC m=+1138.067562126" watchObservedRunningTime="2026-09-08 18:42:43.617473381 +0000 UTC m=+1138.069048514"
Sep 08 18:42:44.205872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:44.205837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:44.210786 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:44.210760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:44.601027 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:44.600942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:42:44.601841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:42:44.601820    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:43:03.714028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.713988    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerID="00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413" exitCode=0
Sep 08 18:43:03.714028 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.714015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerDied","Data":"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"}
Sep 08 18:43:03.714479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.714430    2582 scope.go:117] "RemoveContainer" containerID="00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"
Sep 08 18:43:03.715536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.715504    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:03.715639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.715591    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:03.715701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:03.715636    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:04.721170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:04.721139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerStarted","Data":"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"}
Sep 08 18:43:05.205824 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:05.205777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:43:05.206140 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:05.206115    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 08 18:43:05.206232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:05.206175    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 08 18:43:06.205946 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:06.205909    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 08 18:43:06.206342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:06.205975    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 08 18:43:07.206657 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:07.206566    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 08 18:43:07.206657 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:07.206628    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 08 18:43:08.206748 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:08.206686    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 08 18:43:08.207272 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:08.206768    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 08 18:43:09.206085 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:09.206043    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 08 18:43:09.206273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:09.206104    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 08 18:43:10.210177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:10.210143    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:43:10.210585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:10.210414    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:43:10.211151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:10.211134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:43:11.246931 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.246892    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7575-5cf628"]
Sep 08 18:43:11.249416 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.249396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:11.252362 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.252341    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhbkz\""
Sep 08 18:43:11.257580 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.257556    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-5cf628"]
Sep 08 18:43:11.342887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.342844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv46l\" (UniqueName: \"kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l\") pod \"test-curl-7575-5cf628\" (UID: \"ab459abd-562a-417c-81a6-5a17a301df03\") " pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:11.444158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.444126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv46l\" (UniqueName: \"kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l\") pod \"test-curl-7575-5cf628\" (UID: \"ab459abd-562a-417c-81a6-5a17a301df03\") " pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:11.453757 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.453695    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv46l\" (UniqueName: \"kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l\") pod \"test-curl-7575-5cf628\" (UID: \"ab459abd-562a-417c-81a6-5a17a301df03\") " pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:11.560584 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.560508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:11.633562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.633125    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:43:11.636208 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.636191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.639304 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.639282    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"kube-root-ca.crt\""
Sep 08 18:43:11.639495 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.639289    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"default-dockercfg-f5j7z\""
Sep 08 18:43:11.640789 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.640767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"auth-test-model-792c35-kserve-self-signed-certs\""
Sep 08 18:43:11.640886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.640822    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"openshift-service-ca.crt\""
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.647892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.647937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.648012    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.648043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.648135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.648254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.648165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wwk\" (UniqueName: \"kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.651625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.651599    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:43:11.748786 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.748971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wwk\" (UniqueName: \"kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.748971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.748971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.748971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.748971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.748958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.749319 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.749296    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.749386 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.749338    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.749427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.749391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.751212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.751184    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.751412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.751396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.770833 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.770772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wwk\" (UniqueName: \"kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk\") pod \"auth-test-model-792c35-kserve-58bbc4fff9-xdp8f\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") " pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:11.834365 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.834335    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-5cf628"]
Sep 08 18:43:11.836297 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:11.836272    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab459abd_562a_417c_81a6_5a17a301df03.slice/crio-4a65353d07dd5c831016a3e47933780b17a447eb1d741253561ac74f10e3cb7c WatchSource:0}: Error finding container 4a65353d07dd5c831016a3e47933780b17a447eb1d741253561ac74f10e3cb7c: Status 404 returned error can't find the container with id 4a65353d07dd5c831016a3e47933780b17a447eb1d741253561ac74f10e3cb7c
Sep 08 18:43:11.950712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:11.950673    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:12.229978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.229953    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:43:12.232346 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:12.232318    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08524de7_38db_447b_b015_462e3a68337b.slice/crio-a4b6b8e9dfd579a08bb601cdf43de20d9eada3069a312e1094bf896ac765ea55 WatchSource:0}: Error finding container a4b6b8e9dfd579a08bb601cdf43de20d9eada3069a312e1094bf896ac765ea55: Status 404 returned error can't find the container with id a4b6b8e9dfd579a08bb601cdf43de20d9eada3069a312e1094bf896ac765ea55
Sep 08 18:43:12.773448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.773411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" event={"ID":"08524de7-38db-447b-b015-462e3a68337b","Type":"ContainerStarted","Data":"16c73ef9c1838150727c9b3fb178e30e22fab9e8788ba1311fbadf3c14f6a62d"}
Sep 08 18:43:12.773448 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.773451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" event={"ID":"08524de7-38db-447b-b015-462e3a68337b","Type":"ContainerStarted","Data":"a4b6b8e9dfd579a08bb601cdf43de20d9eada3069a312e1094bf896ac765ea55"}
Sep 08 18:43:12.775012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.774987    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-5cf628" event={"ID":"ab459abd-562a-417c-81a6-5a17a301df03","Type":"ContainerStarted","Data":"f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b"}
Sep 08 18:43:12.775092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.775023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-5cf628" event={"ID":"ab459abd-562a-417c-81a6-5a17a301df03","Type":"ContainerStarted","Data":"4a65353d07dd5c831016a3e47933780b17a447eb1d741253561ac74f10e3cb7c"}
Sep 08 18:43:12.793577 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.793523    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" podStartSLOduration=1.793508909 podStartE2EDuration="1.793508909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:12.791774807 +0000 UTC m=+1167.243349941" watchObservedRunningTime="2026-09-08 18:43:12.793508909 +0000 UTC m=+1167.245084035"
Sep 08 18:43:12.809619 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:12.809569    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7575-5cf628" podStartSLOduration=1.809553251 podStartE2EDuration="1.809553251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:12.809184404 +0000 UTC m=+1167.260759537" watchObservedRunningTime="2026-09-08 18:43:12.809553251 +0000 UTC m=+1167.261128383"
Sep 08 18:43:13.429263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.429219    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7575-5cf628"]
Sep 08 18:43:13.433530 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.433501    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7575-5cf628"]
Sep 08 18:43:13.561110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.561021    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7575-580841"]
Sep 08 18:43:13.561491 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.561473    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab459abd-562a-417c-81a6-5a17a301df03" containerName="test-curl-7575-5cf628"
Sep 08 18:43:13.561578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.561495    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab459abd-562a-417c-81a6-5a17a301df03" containerName="test-curl-7575-5cf628"
Sep 08 18:43:13.561638 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.561616    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab459abd-562a-417c-81a6-5a17a301df03" containerName="test-curl-7575-5cf628"
Sep 08 18:43:13.564574 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.564553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:13.569588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.569564    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-580841"]
Sep 08 18:43:13.667936 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.667900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462kw\" (UniqueName: \"kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw\") pod \"test-curl-7575-580841\" (UID: \"520b37fd-8535-4cb5-b4e1-4bd01d6c1239\") " pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:13.769156 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.769122    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462kw\" (UniqueName: \"kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw\") pod \"test-curl-7575-580841\" (UID: \"520b37fd-8535-4cb5-b4e1-4bd01d6c1239\") " pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:13.778421 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.778386    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-462kw\" (UniqueName: \"kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw\") pod \"test-curl-7575-580841\" (UID: \"520b37fd-8535-4cb5-b4e1-4bd01d6c1239\") " pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:13.780268 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:13.780242    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7575-5cf628" container="test-curl-7575-5cf628"
Sep 08 18:43:13.875198 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:13.875112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:14.148326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:14.148295    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-580841"]
Sep 08 18:43:14.149662 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:14.149629    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod520b37fd_8535_4cb5_b4e1_4bd01d6c1239.slice/crio-dcc027c32f31df1ba709bed3e17d64d58197e55a5f91711b5fc46971e74effa7 WatchSource:0}: Error finding container dcc027c32f31df1ba709bed3e17d64d58197e55a5f91711b5fc46971e74effa7: Status 404 returned error can't find the container with id dcc027c32f31df1ba709bed3e17d64d58197e55a5f91711b5fc46971e74effa7
Sep 08 18:43:14.788114 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:14.788085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-580841" event={"ID":"520b37fd-8535-4cb5-b4e1-4bd01d6c1239","Type":"ContainerStarted","Data":"6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96"}
Sep 08 18:43:14.788482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:14.788127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-580841" event={"ID":"520b37fd-8535-4cb5-b4e1-4bd01d6c1239","Type":"ContainerStarted","Data":"dcc027c32f31df1ba709bed3e17d64d58197e55a5f91711b5fc46971e74effa7"}
Sep 08 18:43:14.788482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:14.788197    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7575-5cf628" podUID="ab459abd-562a-417c-81a6-5a17a301df03" containerName="test-curl-7575-5cf628" containerID="cri-o://f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b" gracePeriod=2
Sep 08 18:43:14.804783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:14.804698    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7575-580841" podStartSLOduration=1.804679283 podStartE2EDuration="1.804679283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:14.803035555 +0000 UTC m=+1169.254610687" watchObservedRunningTime="2026-09-08 18:43:14.804679283 +0000 UTC m=+1169.256254414"
Sep 08 18:43:15.429108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:15.429063    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7575-580841"]
Sep 08 18:43:15.437497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:15.437463    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7575-580841"]
Sep 08 18:43:15.793153 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:15.793119    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7575-580841" container="test-curl-7575-580841"
Sep 08 18:43:16.798858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:16.798815    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7575-580841" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" containerName="test-curl-7575-580841" containerID="cri-o://6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96" gracePeriod=2
Sep 08 18:43:16.944044 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:16.944022    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:17.001280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.001243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv46l\" (UniqueName: \"kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l\") pod \"ab459abd-562a-417c-81a6-5a17a301df03\" (UID: \"ab459abd-562a-417c-81a6-5a17a301df03\") "
Sep 08 18:43:17.003586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.003551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l" (OuterVolumeSpecName: "kube-api-access-bv46l") pod "ab459abd-562a-417c-81a6-5a17a301df03" (UID: "ab459abd-562a-417c-81a6-5a17a301df03"). InnerVolumeSpecName "kube-api-access-bv46l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:43:17.102562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.102475    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv46l\" (UniqueName: \"kubernetes.io/projected/ab459abd-562a-417c-81a6-5a17a301df03-kube-api-access-bv46l\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:43:17.509618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.509582    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7575-334120"]
Sep 08 18:43:17.509991 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.509977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" containerName="test-curl-7575-580841"
Sep 08 18:43:17.510040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.509993    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" containerName="test-curl-7575-580841"
Sep 08 18:43:17.510080 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.510064    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" containerName="test-curl-7575-580841"
Sep 08 18:43:17.513032 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.513016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:17.518541 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.518514    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-334120"]
Sep 08 18:43:17.608079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.608033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgfb\" (UniqueName: \"kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb\") pod \"test-curl-7575-334120\" (UID: \"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9\") " pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:17.708851 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.708806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgfb\" (UniqueName: \"kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb\") pod \"test-curl-7575-334120\" (UID: \"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9\") " pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:17.717683 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.717652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgfb\" (UniqueName: \"kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb\") pod \"test-curl-7575-334120\" (UID: \"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9\") " pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:17.804871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.804788    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab459abd-562a-417c-81a6-5a17a301df03" containerID="f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b" exitCode=137
Sep 08 18:43:17.804871 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.804840    2582 scope.go:117] "RemoveContainer" containerID="f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b"
Sep 08 18:43:17.805357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.804839    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:17.814562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.814540    2582 scope.go:117] "RemoveContainer" containerID="f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b"
Sep 08 18:43:17.814898 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:17.814880    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b\": container with ID starting with f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b not found: ID does not exist" containerID="f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b"
Sep 08 18:43:17.814971 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.814908    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b"} err="failed to get container status \"f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b\": rpc error: code = NotFound desc = could not find container \"f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b\": container with ID starting with f6f7e8a7e5fe04e63181f3274d99fc334d3e24d5239e532cf7255636d80f9e9b not found: ID does not exist"
Sep 08 18:43:17.817133 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:17.817111    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7575-5cf628\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="ab459abd-562a-417c-81a6-5a17a301df03" pod="opendatahub/test-curl-7575-5cf628"
Sep 08 18:43:17.823239 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:17.823217    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:18.096514 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:18.096482    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd1e6f7_cca2_4a9f_88b6_d8be808e67e9.slice/crio-770efbf0ea204a0915bd12c6237d09e0ab6be3a4eebca42ee27ea8ce86b6b1ab WatchSource:0}: Error finding container 770efbf0ea204a0915bd12c6237d09e0ab6be3a4eebca42ee27ea8ce86b6b1ab: Status 404 returned error can't find the container with id 770efbf0ea204a0915bd12c6237d09e0ab6be3a4eebca42ee27ea8ce86b6b1ab
Sep 08 18:43:18.100990 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.100967    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab459abd-562a-417c-81a6-5a17a301df03" path="/var/lib/kubelet/pods/ab459abd-562a-417c-81a6-5a17a301df03/volumes"
Sep 08 18:43:18.101244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.101231    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7575-334120"]
Sep 08 18:43:18.814497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.814463    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-334120" event={"ID":"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9","Type":"ContainerStarted","Data":"81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e"}
Sep 08 18:43:18.814933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.814514    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7575-334120" event={"ID":"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9","Type":"ContainerStarted","Data":"770efbf0ea204a0915bd12c6237d09e0ab6be3a4eebca42ee27ea8ce86b6b1ab"}
Sep 08 18:43:18.830950 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.830890    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7575-334120" podStartSLOduration=1.830872636 podStartE2EDuration="1.830872636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:18.830228828 +0000 UTC m=+1173.281803959" watchObservedRunningTime="2026-09-08 18:43:18.830872636 +0000 UTC m=+1173.282447768"
Sep 08 18:43:18.957575 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:18.957551    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:19.021769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.021716    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-462kw\" (UniqueName: \"kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw\") pod \"520b37fd-8535-4cb5-b4e1-4bd01d6c1239\" (UID: \"520b37fd-8535-4cb5-b4e1-4bd01d6c1239\") "
Sep 08 18:43:19.023839 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.023804    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw" (OuterVolumeSpecName: "kube-api-access-462kw") pod "520b37fd-8535-4cb5-b4e1-4bd01d6c1239" (UID: "520b37fd-8535-4cb5-b4e1-4bd01d6c1239"). InnerVolumeSpecName "kube-api-access-462kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:43:19.123206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.123126    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462kw\" (UniqueName: \"kubernetes.io/projected/520b37fd-8535-4cb5-b4e1-4bd01d6c1239-kube-api-access-462kw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:43:19.476092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.476056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7575-334120"]
Sep 08 18:43:19.480557 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.480530    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7575-334120"]
Sep 08 18:43:19.823016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.822925    2582 generic.go:356] "Generic (PLEG): container finished" podID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" containerID="6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96" exitCode=137
Sep 08 18:43:19.823016 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.822980    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:19.823503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.823030    2582 scope.go:117] "RemoveContainer" containerID="6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96"
Sep 08 18:43:19.823503 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:19.823433    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7575-334120" container="test-curl-7575-334120"
Sep 08 18:43:19.832894 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.832870    2582 scope.go:117] "RemoveContainer" containerID="6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96"
Sep 08 18:43:19.833210 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:19.833189    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96\": container with ID starting with 6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96 not found: ID does not exist" containerID="6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96"
Sep 08 18:43:19.833279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:19.833219    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96"} err="failed to get container status \"6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96\": rpc error: code = NotFound desc = could not find container \"6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96\": container with ID starting with 6a4c693ff0e707d471027ec468f75fe3f3195ae3fc71a6de98d5a1812189dc96 not found: ID does not exist"
Sep 08 18:43:19.834609 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:19.834590    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7575-580841\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" pod="opendatahub/test-curl-7575-580841"
Sep 08 18:43:20.099648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.099566    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520b37fd-8535-4cb5-b4e1-4bd01d6c1239" path="/var/lib/kubelet/pods/520b37fd-8535-4cb5-b4e1-4bd01d6c1239/volumes"
Sep 08 18:43:20.353498 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.353402    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:43:20.353956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.353937    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" containerName="test-curl-7575-334120"
Sep 08 18:43:20.353956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.353958    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" containerName="test-curl-7575-334120"
Sep 08 18:43:20.354087 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.354035    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" containerName="test-curl-7575-334120"
Sep 08 18:43:20.361365 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.361343    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.365196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.365177    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-783195cb-openshift-default-dockercfg-j5d6l\""
Sep 08 18:43:20.372647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.372617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:43:20.436199 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.436144    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5bf\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.437907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.437790    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.538631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.538631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.538631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.538631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5bf\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.538870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.539068    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.539110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.539194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539426 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.539348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.539829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.539798    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.541333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.541309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.541653 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.541632    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.552920 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.552896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5bf\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.553043 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.552898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token\") pod \"e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") " pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.673130 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.673076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:20.830508 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.830453    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7575-334120" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" containerName="test-curl-7575-334120" containerID="cri-o://81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e" gracePeriod=2
Sep 08 18:43:20.955347 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.955318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:43:20.962496 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.962461    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:20.962630 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.962537    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:20.962630 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:20.962573    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:21.837651 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.837618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerStarted","Data":"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"}
Sep 08 18:43:21.838042 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.837669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerStarted","Data":"5daa8ddab71c61583898a804ed813ad41523fc9902518ba24bcbed627221d7af"}
Sep 08 18:43:21.861650 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.861589    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podStartSLOduration=1.861574213 podStartE2EDuration="1.861574213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:21.860005574 +0000 UTC m=+1176.311580719" watchObservedRunningTime="2026-09-08 18:43:21.861574213 +0000 UTC m=+1176.313149345"
Sep 08 18:43:21.951080 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.951038    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:21.951280 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.951114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:21.964024 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:21.963997    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:22.673678 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:22.673631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:22.679228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:22.679203    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:22.842497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:22.842465    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:22.843625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:22.843602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:22.854170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:22.854139    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:43:23.019993 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.019970    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:23.165643 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.165605    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brgfb\" (UniqueName: \"kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb\") pod \"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9\" (UID: \"0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9\") "
Sep 08 18:43:23.167921 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.167890    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb" (OuterVolumeSpecName: "kube-api-access-brgfb") pod "0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" (UID: "0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9"). InnerVolumeSpecName "kube-api-access-brgfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:43:23.267397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.267307    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgfb\" (UniqueName: \"kubernetes.io/projected/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9-kube-api-access-brgfb\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:43:23.848939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.848905    2582 generic.go:356] "Generic (PLEG): container finished" podID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" containerID="81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e" exitCode=137
Sep 08 18:43:23.849436 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.848950    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:23.849436 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.849005    2582 scope.go:117] "RemoveContainer" containerID="81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e"
Sep 08 18:43:23.859957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.859938    2582 scope.go:117] "RemoveContainer" containerID="81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e"
Sep 08 18:43:23.860279 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:23.860259    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e\": container with ID starting with 81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e not found: ID does not exist" containerID="81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e"
Sep 08 18:43:23.860331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:23.860288    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e"} err="failed to get container status \"81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e\": rpc error: code = NotFound desc = could not find container \"81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e\": container with ID starting with 81df36d1623ad0c941866f5c4f418d25f878573501f328c576880765d7a6dd5e not found: ID does not exist"
Sep 08 18:43:23.860983 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:43:23.860962    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7575-334120\" is forbidden: User \"system:node:ip-10-0-142-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-80.ec2.internal' and this object" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" pod="opendatahub/test-curl-7575-334120"
Sep 08 18:43:24.100062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:24.099984    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9" path="/var/lib/kubelet/pods/0fd1e6f7-cca2-4a9f-88b6-d8be808e67e9/volumes"
Sep 08 18:43:40.114425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.114389    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:43:40.126819 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.126785    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.130293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.129797    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:43:40.130293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.130104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"openshift-service-ca.crt\""
Sep 08 18:43:40.131463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.131440    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"kube-root-ca.crt\""
Sep 08 18:43:40.132267 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.131617    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"auth-test-model-6255f2-kserve-self-signed-certs\""
Sep 08 18:43:40.132454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.131671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"default-dockercfg-l4kvl\""
Sep 08 18:43:40.242000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.241898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.242000 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.241951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.242210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.242025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.242210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.242094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.242210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.242138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.242210 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.242209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7cr\" (UniqueName: \"kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343053    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7cr\" (UniqueName: \"kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.343647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.343623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.345552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.345528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.345796 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.345780    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.352571 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.352545    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7cr\" (UniqueName: \"kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr\") pod \"auth-test-model-6255f2-kserve-59b6984f95-rbb4r\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.440869 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.440835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:40.721082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.721049    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:43:40.722311 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:40.722281    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7c7c99c_41ba_462b_a28a_c3699e8df8a3.slice/crio-1330c1d36b535dfa01998ff211269a8370690b55cb73740117d6882834016732 WatchSource:0}: Error finding container 1330c1d36b535dfa01998ff211269a8370690b55cb73740117d6882834016732: Status 404 returned error can't find the container with id 1330c1d36b535dfa01998ff211269a8370690b55cb73740117d6882834016732
Sep 08 18:43:40.944926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.944886    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" event={"ID":"f7c7c99c-41ba-462b-a28a-c3699e8df8a3","Type":"ContainerStarted","Data":"b30ff37bbeddfecdb64f6c2bb39920f44ee24f221a95086edb1042abaf59819c"}
Sep 08 18:43:40.945121 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.944935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" event={"ID":"f7c7c99c-41ba-462b-a28a-c3699e8df8a3","Type":"ContainerStarted","Data":"1330c1d36b535dfa01998ff211269a8370690b55cb73740117d6882834016732"}
Sep 08 18:43:40.963805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:40.963741    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" podStartSLOduration=0.963707486 podStartE2EDuration="963.707486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:40.963270846 +0000 UTC m=+1195.414845980" watchObservedRunningTime="2026-09-08 18:43:40.963707486 +0000 UTC m=+1195.415282615"
Sep 08 18:43:42.958208 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.958171    2582 generic.go:356] "Generic (PLEG): container finished" podID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerID="e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc" exitCode=0
Sep 08 18:43:42.958718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.958221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerDied","Data":"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"}
Sep 08 18:43:42.958718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.958616    2582 scope.go:117] "RemoveContainer" containerID="e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"
Sep 08 18:43:42.959868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.959840    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:42.959954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.959910    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:42.959954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:42.959941    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:43:43.966074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:43.966041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerStarted","Data":"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"}
Sep 08 18:43:44.674079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:44.674044    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:44.674454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:44.674431    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:44.674606 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:44.674581    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:45.673953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:45.673901    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:45.674467 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:45.673997    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:46.674615 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:46.674561    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:46.675147 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:46.674641    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:47.673612 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:47.673571    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:47.673939 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:47.673905    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:48.674493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:48.674452    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:48.675158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:48.675122    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:49.674425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:49.674378    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 08 18:43:49.674611 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:49.674469    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 08 18:43:50.441008 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.440976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:50.441206 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.441027    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:50.453688 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.453652    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:50.677531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.677500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:50.677964 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.677774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:50.678455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:50.678437    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:43:51.024633 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.024605    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:43:51.227082 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.227049    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:43:51.231917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.231899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.235173 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.235151    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-783195cb\"/\"e2e-lifecycle-model-783195cb-kserve-self-signed-certs\""
Sep 08 18:43:51.235298 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.235161    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-783195cb\"/\"default-dockercfg-n62k4\""
Sep 08 18:43:51.235512 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.235485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-783195cb\"/\"openshift-service-ca.crt\""
Sep 08 18:43:51.235794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.235779    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-783195cb\"/\"kube-root-ca.crt\""
Sep 08 18:43:51.242752 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.242701    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:43:51.357007 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.356904    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.357007 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.356957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.357236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.357059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.357236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.357147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xsf\" (UniqueName: \"kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.357236 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.357216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.357361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.357234    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.457880 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.457840    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.457919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xsf\" (UniqueName: \"kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.457965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.457981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.458010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458112 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.458048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.458348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.458376    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.458476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.458402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.460399 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.460378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.460681 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.460664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.467624 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.467593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xsf\" (UniqueName: \"kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf\") pod \"e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") " pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.543205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.543147    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:43:51.816492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:51.816468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:43:51.817975 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:43:51.817944    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a5833e4_30d9_4a17_aba8_251340cdc032.slice/crio-e8376d65bec9d820baf895a48eb50c3c0824fd75b12fe56f8816729bd175ac5d WatchSource:0}: Error finding container e8376d65bec9d820baf895a48eb50c3c0824fd75b12fe56f8816729bd175ac5d: Status 404 returned error can't find the container with id e8376d65bec9d820baf895a48eb50c3c0824fd75b12fe56f8816729bd175ac5d
Sep 08 18:43:52.020178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:52.020134    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" event={"ID":"3a5833e4-30d9-4a17-aba8-251340cdc032","Type":"ContainerStarted","Data":"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"}
Sep 08 18:43:52.020374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:52.020186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" event={"ID":"3a5833e4-30d9-4a17-aba8-251340cdc032","Type":"ContainerStarted","Data":"e8376d65bec9d820baf895a48eb50c3c0824fd75b12fe56f8816729bd175ac5d"}
Sep 08 18:43:52.041573 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:43:52.041515    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" podStartSLOduration=1.041496705 podStartE2EDuration="1.041496705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:52.039098283 +0000 UTC m=+1206.490673414" watchObservedRunningTime="2026-09-08 18:43:52.041496705 +0000 UTC m=+1206.493071837"
Sep 08 18:44:01.543869 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:01.543815    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:01.544325 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:01.543914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:01.557124 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:01.557098    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:02.083907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:02.083879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:41.752292 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:41.752206    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:44:46.194685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:46.194655    2582 scope.go:117] "RemoveContainer" containerID="0e896ec988eb531e59a9c6bcec73c71291828bdba8740b0b0765a4a4922f9905"
Sep 08 18:44:56.772765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:56.772682    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" podUID="3a5833e4-30d9-4a17-aba8-251340cdc032" containerName="main" containerID="cri-o://fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7" gracePeriod=45
Sep 08 18:44:57.033207 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.033146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:57.107361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107331    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107389    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107458    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107673 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86xsf\" (UniqueName: \"kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107673 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm\") pod \"3a5833e4-30d9-4a17-aba8-251340cdc032\" (UID: \"3a5833e4-30d9-4a17-aba8-251340cdc032\") "
Sep 08 18:44:57.107673 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107641    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home" (OuterVolumeSpecName: "home") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:44:57.107806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.107645    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:44:57.108132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.108075    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.108132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.108099    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.108132 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.108075    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:44:57.110169 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.110129    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf" (OuterVolumeSpecName: "kube-api-access-86xsf") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "kube-api-access-86xsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:44:57.110505 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.110479    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:44:57.110599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.110516    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm" (OuterVolumeSpecName: "dshm") pod "3a5833e4-30d9-4a17-aba8-251340cdc032" (UID: "3a5833e4-30d9-4a17-aba8-251340cdc032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:44:57.209090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.209054    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a5833e4-30d9-4a17-aba8-251340cdc032-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.209090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.209087    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xsf\" (UniqueName: \"kubernetes.io/projected/3a5833e4-30d9-4a17-aba8-251340cdc032-kube-api-access-86xsf\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.209090 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.209096    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.209329 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.209105    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a5833e4-30d9-4a17-aba8-251340cdc032-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:44:57.368882 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.368783    2582 generic.go:356] "Generic (PLEG): container finished" podID="3a5833e4-30d9-4a17-aba8-251340cdc032" containerID="fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7" exitCode=0
Sep 08 18:44:57.368882 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.368845    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"
Sep 08 18:44:57.368882 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.368868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" event={"ID":"3a5833e4-30d9-4a17-aba8-251340cdc032","Type":"ContainerDied","Data":"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"}
Sep 08 18:44:57.369153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.368913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz" event={"ID":"3a5833e4-30d9-4a17-aba8-251340cdc032","Type":"ContainerDied","Data":"e8376d65bec9d820baf895a48eb50c3c0824fd75b12fe56f8816729bd175ac5d"}
Sep 08 18:44:57.369153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.368930    2582 scope.go:117] "RemoveContainer" containerID="fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"
Sep 08 18:44:57.378496 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.378480    2582 scope.go:117] "RemoveContainer" containerID="fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"
Sep 08 18:44:57.378789 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:44:57.378766    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7\": container with ID starting with fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7 not found: ID does not exist" containerID="fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"
Sep 08 18:44:57.378841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.378802    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7"} err="failed to get container status \"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7\": rpc error: code = NotFound desc = could not find container \"fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7\": container with ID starting with fa5e6b543749037fe5984301f4a7115623200d39d7048c96cf369bb1aea819f7 not found: ID does not exist"
Sep 08 18:44:57.391449 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.391424    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:44:57.396130 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:57.396104    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-783195cb/e2e-lifecycle-model-783195cb-kserve-dc7d899d7-qx2gz"]
Sep 08 18:44:58.099426 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:44:58.099388    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5833e4-30d9-4a17-aba8-251340cdc032" path="/var/lib/kubelet/pods/3a5833e4-30d9-4a17-aba8-251340cdc032/volumes"
Sep 08 18:45:00.148252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.148220    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"]
Sep 08 18:45:00.148809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.148614    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5833e4-30d9-4a17-aba8-251340cdc032" containerName="main"
Sep 08 18:45:00.148809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.148626    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5833e4-30d9-4a17-aba8-251340cdc032" containerName="main"
Sep 08 18:45:00.148809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.148739    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5833e4-30d9-4a17-aba8-251340cdc032" containerName="main"
Sep 08 18:45:00.153072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.153055    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"]
Sep 08 18:45:00.153277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.153261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"
Sep 08 18:45:00.156121 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.156096    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:45:00.156383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.156367    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"
Sep 08 18:45:00.163901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.163883    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"]
Sep 08 18:45:00.166827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.166805    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"]
Sep 08 18:45:00.167425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.167383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"
Sep 08 18:45:00.177477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.177453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"
Sep 08 18:45:00.449948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.449765    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"]
Sep 08 18:45:00.453125 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:45:00.453098    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d8d1f6_b3cd_4d49_ba7f_39eebc765b86.slice/crio-f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf WatchSource:0}: Error finding container f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf: Status 404 returned error can't find the container with id f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf
Sep 08 18:45:00.466431 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:00.466406    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"]
Sep 08 18:45:00.469358 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:45:00.469299    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2d7849_87e0_4e63_b87f_d1945a2666f2.slice/crio-2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe WatchSource:0}: Error finding container 2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe: Status 404 returned error can't find the container with id 2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe
Sep 08 18:45:01.393388 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.393358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr" event={"ID":"b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86","Type":"ContainerStarted","Data":"2c72ea1f0b45fc1027352c19935ed93a9bf446aad0082bc0cd0a9cfa08d4e2b7"}
Sep 08 18:45:01.393787 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.393452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr" event={"ID":"b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86","Type":"ContainerStarted","Data":"f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf"}
Sep 08 18:45:01.394854 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.394828    2582 generic.go:356] "Generic (PLEG): container finished" podID="fd2d7849-87e0-4e63-b87f-d1945a2666f2" containerID="97435763a3ad7f19e0791a024438758ab68112230eeeed19cbc1fe47b99054d5" exitCode=0
Sep 08 18:45:01.394941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.394903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf" event={"ID":"fd2d7849-87e0-4e63-b87f-d1945a2666f2","Type":"ContainerDied","Data":"97435763a3ad7f19e0791a024438758ab68112230eeeed19cbc1fe47b99054d5"}
Sep 08 18:45:01.394981 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.394948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf" event={"ID":"fd2d7849-87e0-4e63-b87f-d1945a2666f2","Type":"ContainerStarted","Data":"2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe"}
Sep 08 18:45:01.414134 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:01.414083    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr" podStartSLOduration=1.414070202 podStartE2EDuration="1.414070202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:45:01.412386875 +0000 UTC m=+1275.863962008" watchObservedRunningTime="2026-09-08 18:45:01.414070202 +0000 UTC m=+1275.865645335"
Sep 08 18:45:02.402392 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:02.402356    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86" containerID="2c72ea1f0b45fc1027352c19935ed93a9bf446aad0082bc0cd0a9cfa08d4e2b7" exitCode=0
Sep 08 18:45:02.402874 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:02.402445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr" event={"ID":"b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86","Type":"ContainerDied","Data":"2c72ea1f0b45fc1027352c19935ed93a9bf446aad0082bc0cd0a9cfa08d4e2b7"}
Sep 08 18:45:02.556989 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:02.556965    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"
Sep 08 18:45:03.409309 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:03.409277    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"
Sep 08 18:45:03.409695 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:03.409312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf" event={"ID":"fd2d7849-87e0-4e63-b87f-d1945a2666f2","Type":"ContainerDied","Data":"2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe"}
Sep 08 18:45:03.409695 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:03.409358    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab99930fdb79763eff7cdf1c198e704954bdee261cfe281e78fea345421ccbe"
Sep 08 18:45:03.549254 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:03.549232    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"
Sep 08 18:45:04.415914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:04.415882    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr"
Sep 08 18:45:04.416290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:04.415920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814885-cllpr" event={"ID":"b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86","Type":"ContainerDied","Data":"f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf"}
Sep 08 18:45:04.416290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:04.415969    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7bda6f5f76ebf6e59d60a8a7ac261b64b28e7b8e37d2fee194a754d162946cf"
Sep 08 18:45:30.878794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:30.878756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:45:30.879218 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:30.879021    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" containerID="cri-o://8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a" gracePeriod=30
Sep 08 18:45:35.680457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:35.680407    2582 patch_prober.go:28] interesting pod/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 18:45:35.680994 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:35.680481    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:45:36.140957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.140932    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:45:36.198848 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.198848 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198851    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd5bf\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198935    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.198986    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199058 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199032    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199313 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199102    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199313 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199140    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data\") pod \"d70bd283-9c9b-444b-8161-b07dcc5746fc\" (UID: \"d70bd283-9c9b-444b-8161-b07dcc5746fc\") "
Sep 08 18:45:36.199313 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199208    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:45:36.199476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199333    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:45:36.199476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:45:36.199594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199555    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:45:36.199594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199582    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.199700 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199601    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.199700 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199616    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.199700 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.199639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:45:36.201208 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.201183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:45:36.201291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.201270    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:45:36.201783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.201757    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf" (OuterVolumeSpecName: "kube-api-access-kd5bf") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "kube-api-access-kd5bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:45:36.201922 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.201838    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d70bd283-9c9b-444b-8161-b07dcc5746fc" (UID: "d70bd283-9c9b-444b-8161-b07dcc5746fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300402    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300430    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5bf\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-kube-api-access-kd5bf\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300440    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300450    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300458    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d70bd283-9c9b-444b-8161-b07dcc5746fc-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.300484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.300465    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d70bd283-9c9b-444b-8161-b07dcc5746fc-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:45:36.596621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.596530    2582 generic.go:356] "Generic (PLEG): container finished" podID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerID="8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a" exitCode=0
Sep 08 18:45:36.596621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.596581    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"
Sep 08 18:45:36.596901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.596613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerDied","Data":"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"}
Sep 08 18:45:36.596901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.596678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr" event={"ID":"d70bd283-9c9b-444b-8161-b07dcc5746fc","Type":"ContainerDied","Data":"5daa8ddab71c61583898a804ed813ad41523fc9902518ba24bcbed627221d7af"}
Sep 08 18:45:36.596901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.596701    2582 scope.go:117] "RemoveContainer" containerID="8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"
Sep 08 18:45:36.606995 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.606977    2582 scope.go:117] "RemoveContainer" containerID="e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"
Sep 08 18:45:36.615010 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.614992    2582 scope.go:117] "RemoveContainer" containerID="8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"
Sep 08 18:45:36.615303 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:45:36.615284    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a\": container with ID starting with 8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a not found: ID does not exist" containerID="8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"
Sep 08 18:45:36.615364 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.615312    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a"} err="failed to get container status \"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a\": rpc error: code = NotFound desc = could not find container \"8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a\": container with ID starting with 8299173ed6726c1997da0751d8df5ded2fdc03c5464bf41144e07d6c82c2f43a not found: ID does not exist"
Sep 08 18:45:36.615364 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.615332    2582 scope.go:117] "RemoveContainer" containerID="e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"
Sep 08 18:45:36.615579 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:45:36.615562    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc\": container with ID starting with e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc not found: ID does not exist" containerID="e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"
Sep 08 18:45:36.615634 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.615591    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc"} err="failed to get container status \"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc\": rpc error: code = NotFound desc = could not find container \"e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc\": container with ID starting with e11eeb44ffe58e0492989668dab78000c27c57bbd04b05d4da232a355f46f2dc not found: ID does not exist"
Sep 08 18:45:36.626601 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.626577    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:45:36.635619 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:36.635599    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-783195cb-openshift-default-6bf4bbfd7b-w2bjr"]
Sep 08 18:45:38.099188 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:38.099155    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" path="/var/lib/kubelet/pods/d70bd283-9c9b-444b-8161-b07dcc5746fc/volumes"
Sep 08 18:45:46.215349 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:46.215317    2582 scope.go:117] "RemoveContainer" containerID="46e82124d8408b2435423ef9007bf82e9e40c7f801a785f7e492f865eab11b75"
Sep 08 18:45:46.224215 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:46.224195    2582 scope.go:117] "RemoveContainer" containerID="433fe10f0c4dae3b5c454729fa13b9cf24c0373c56c441c994cca5a80e7d6c4d"
Sep 08 18:45:46.234509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:46.234485    2582 scope.go:117] "RemoveContainer" containerID="eee451810923d87a6f051113a46c68725c9a7c6af6ca1bfbebba7a5624dafda6"
Sep 08 18:45:51.610312 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:51.610280    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:45:52.515805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:45:52.515770    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:46:02.029867 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.029825    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:46:02.030260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030243    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:02.030260 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030255    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:02.030339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030263    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86" containerName="cleanup"
Sep 08 18:46:02.030339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030268    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86" containerName="cleanup"
Sep 08 18:46:02.030339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030290    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2d7849-87e0-4e63-b87f-d1945a2666f2" containerName="cleanup"
Sep 08 18:46:02.030339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030296    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2d7849-87e0-4e63-b87f-d1945a2666f2" containerName="cleanup"
Sep 08 18:46:02.030518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030367    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2d7849-87e0-4e63-b87f-d1945a2666f2" containerName="cleanup"
Sep 08 18:46:02.030518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030376    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86" containerName="cleanup"
Sep 08 18:46:02.030518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030382    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:02.030518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.030387    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:02.035101 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.035079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.038773 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.038747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"sub-test-model-792c35-kserve-self-signed-certs\""
Sep 08 18:46:02.047578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.047555    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:46:02.155305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.155519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155353    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2k88\" (UniqueName: \"kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.155519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.155519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.155519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.155710 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.155533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256311    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2k88\" (UniqueName: \"kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256429 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256615 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256615 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.256881 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.257014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.257014 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.256979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.259263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.259233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.259367 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.259300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.266484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.266455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2k88\" (UniqueName: \"kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88\") pod \"sub-test-model-792c35-kserve-6494d84bf-wbvvf\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") " pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.344322 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.344222    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:02.619328 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.619297    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:46:02.621145 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:46:02.621115    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32decf86_6654_497f_b5ac_1fa579466a1d.slice/crio-4bdecd3eaefafa3f89babb9739943188ad5feed5769ae74ebd457842d56255b2 WatchSource:0}: Error finding container 4bdecd3eaefafa3f89babb9739943188ad5feed5769ae74ebd457842d56255b2: Status 404 returned error can't find the container with id 4bdecd3eaefafa3f89babb9739943188ad5feed5769ae74ebd457842d56255b2
Sep 08 18:46:02.736195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.736153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" event={"ID":"32decf86-6654-497f-b5ac-1fa579466a1d","Type":"ContainerStarted","Data":"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"}
Sep 08 18:46:02.736195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.736201    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" event={"ID":"32decf86-6654-497f-b5ac-1fa579466a1d","Type":"ContainerStarted","Data":"4bdecd3eaefafa3f89babb9739943188ad5feed5769ae74ebd457842d56255b2"}
Sep 08 18:46:02.758534 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:02.758469    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" podStartSLOduration=0.758447443 podStartE2EDuration="758.447443ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:02.755631926 +0000 UTC m=+1337.207207064" watchObservedRunningTime="2026-09-08 18:46:02.758447443 +0000 UTC m=+1337.210022651"
Sep 08 18:46:06.630886 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.630818    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" podUID="08524de7-38db-447b-b015-462e3a68337b" containerName="main" containerID="cri-o://16c73ef9c1838150727c9b3fb178e30e22fab9e8788ba1311fbadf3c14f6a62d" gracePeriod=45
Sep 08 18:46:06.758745 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.758682    2582 generic.go:356] "Generic (PLEG): container finished" podID="08524de7-38db-447b-b015-462e3a68337b" containerID="16c73ef9c1838150727c9b3fb178e30e22fab9e8788ba1311fbadf3c14f6a62d" exitCode=0
Sep 08 18:46:06.758924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.758772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" event={"ID":"08524de7-38db-447b-b015-462e3a68337b","Type":"ContainerDied","Data":"16c73ef9c1838150727c9b3fb178e30e22fab9e8788ba1311fbadf3c14f6a62d"}
Sep 08 18:46:06.889154 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.889133    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:46:06.901264 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901373 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4wwk\" (UniqueName: \"kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901373 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901354    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901374    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901422    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm\") pod \"08524de7-38db-447b-b015-462e3a68337b\" (UID: \"08524de7-38db-447b-b015-462e3a68337b\") "
Sep 08 18:46:06.901588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache" (OuterVolumeSpecName: "model-cache") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:06.901637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901584    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home" (OuterVolumeSpecName: "home") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:06.901637 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901625    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:06.901828 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901807    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:06.901828 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901827    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:06.901988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.901836    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:06.903776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.903701    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:46:06.903866 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.903798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm" (OuterVolumeSpecName: "dshm") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:06.903927 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:06.903913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk" (OuterVolumeSpecName: "kube-api-access-b4wwk") pod "08524de7-38db-447b-b015-462e3a68337b" (UID: "08524de7-38db-447b-b015-462e3a68337b"). InnerVolumeSpecName "kube-api-access-b4wwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:46:07.003077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.003041    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4wwk\" (UniqueName: \"kubernetes.io/projected/08524de7-38db-447b-b015-462e3a68337b-kube-api-access-b4wwk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.003077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.003072    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08524de7-38db-447b-b015-462e3a68337b-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.003077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.003083    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08524de7-38db-447b-b015-462e3a68337b-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.535916 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.535864    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" podUID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" containerName="main" containerID="cri-o://b30ff37bbeddfecdb64f6c2bb39920f44ee24f221a95086edb1042abaf59819c" gracePeriod=45
Sep 08 18:46:07.766989 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.766954    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" containerID="b30ff37bbeddfecdb64f6c2bb39920f44ee24f221a95086edb1042abaf59819c" exitCode=0
Sep 08 18:46:07.767528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.767023    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" event={"ID":"f7c7c99c-41ba-462b-a28a-c3699e8df8a3","Type":"ContainerDied","Data":"b30ff37bbeddfecdb64f6c2bb39920f44ee24f221a95086edb1042abaf59819c"}
Sep 08 18:46:07.768523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.768504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f" event={"ID":"08524de7-38db-447b-b015-462e3a68337b","Type":"ContainerDied","Data":"a4b6b8e9dfd579a08bb601cdf43de20d9eada3069a312e1094bf896ac765ea55"}
Sep 08 18:46:07.768629 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.768543    2582 scope.go:117] "RemoveContainer" containerID="16c73ef9c1838150727c9b3fb178e30e22fab9e8788ba1311fbadf3c14f6a62d"
Sep 08 18:46:07.768629 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.768545    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"
Sep 08 18:46:07.794675 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.794654    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:46:07.805563 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.805538    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:46:07.810700 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf7cr\" (UniqueName: \"kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.810786 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810716    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.810890 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810867    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.810975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810961    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.810975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810965    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:07.811048 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.810997    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.811095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811070    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs\") pod \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\" (UID: \"f7c7c99c-41ba-462b-a28a-c3699e8df8a3\") "
Sep 08 18:46:07.811228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home" (OuterVolumeSpecName: "home") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:07.811333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:07.811415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811400    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.811482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811420    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.811482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.811431    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.813413 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.813387    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm" (OuterVolumeSpecName: "dshm") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:46:07.813484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.813469    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:46:07.813530 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.813482    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr" (OuterVolumeSpecName: "kube-api-access-qf7cr") pod "f7c7c99c-41ba-462b-a28a-c3699e8df8a3" (UID: "f7c7c99c-41ba-462b-a28a-c3699e8df8a3"). InnerVolumeSpecName "kube-api-access-qf7cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:46:07.815278 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.815260    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/auth-test-model-792c35-kserve-58bbc4fff9-xdp8f"]
Sep 08 18:46:07.911923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.911895    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.911923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.911920    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7cr\" (UniqueName: \"kubernetes.io/projected/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-kube-api-access-qf7cr\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:07.911923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:07.911931    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c7c99c-41ba-462b-a28a-c3699e8df8a3-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:46:08.098972 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.098889    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08524de7-38db-447b-b015-462e3a68337b" path="/var/lib/kubelet/pods/08524de7-38db-447b-b015-462e3a68337b/volumes"
Sep 08 18:46:08.775225 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.775193    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"
Sep 08 18:46:08.775225 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.775215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r" event={"ID":"f7c7c99c-41ba-462b-a28a-c3699e8df8a3","Type":"ContainerDied","Data":"1330c1d36b535dfa01998ff211269a8370690b55cb73740117d6882834016732"}
Sep 08 18:46:08.775782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.775265    2582 scope.go:117] "RemoveContainer" containerID="b30ff37bbeddfecdb64f6c2bb39920f44ee24f221a95086edb1042abaf59819c"
Sep 08 18:46:08.793991 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.793966    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:46:08.795875 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:08.795855    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/auth-test-model-6255f2-kserve-59b6984f95-rbb4r"]
Sep 08 18:46:10.098815 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:10.098785    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" path="/var/lib/kubelet/pods/f7c7c99c-41ba-462b-a28a-c3699e8df8a3/volumes"
Sep 08 18:46:12.344709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:12.344673    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:12.345193 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:12.344846    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:12.357412 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:12.357390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:12.810044 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:12.810014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:46:21.714369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:46:21.714781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714768    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:21.714832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714783    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70bd283-9c9b-444b-8161-b07dcc5746fc" containerName="istio-proxy"
Sep 08 18:46:21.714832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714799    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08524de7-38db-447b-b015-462e3a68337b" containerName="main"
Sep 08 18:46:21.714832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714804    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="08524de7-38db-447b-b015-462e3a68337b" containerName="main"
Sep 08 18:46:21.714832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" containerName="main"
Sep 08 18:46:21.714832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714821    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" containerName="main"
Sep 08 18:46:21.714982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714874    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="08524de7-38db-447b-b015-462e3a68337b" containerName="main"
Sep 08 18:46:21.714982 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.714886    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c7c99c-41ba-462b-a28a-c3699e8df8a3" containerName="main"
Sep 08 18:46:21.717973 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.717954    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.722173 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.722149    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"kube-root-ca.crt\""
Sep 08 18:46:21.722308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.722172    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"sub-test-model-6255f2-kserve-self-signed-certs\""
Sep 08 18:46:21.722308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.722199    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"openshift-service-ca.crt\""
Sep 08 18:46:21.722308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.722224    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"default-dockercfg-l4kvl\""
Sep 08 18:46:21.727704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.727685    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:46:21.850517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.850709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.850709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.850709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.850862 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.850862 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.850841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggsh\" (UniqueName: \"kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggsh\" (UniqueName: \"kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952569 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952569 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.952696 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.952680    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.954613 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.954592    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.954791 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.954772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:21.961625 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:21.961597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggsh\" (UniqueName: \"kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh\") pod \"sub-test-model-6255f2-kserve-699489bb4c-749zn\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:22.028783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:22.028697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:22.299975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:22.299951    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:46:22.301818 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:46:22.301788    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5d66550_e67b_4753_9305_078172a6c5b5.slice/crio-f550899c99a0cc3ac7d3328bae96ba643e3f4a4e77503bbcb111b82dd2e17fce WatchSource:0}: Error finding container f550899c99a0cc3ac7d3328bae96ba643e3f4a4e77503bbcb111b82dd2e17fce: Status 404 returned error can't find the container with id f550899c99a0cc3ac7d3328bae96ba643e3f4a4e77503bbcb111b82dd2e17fce
Sep 08 18:46:22.851710 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:22.851673    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" event={"ID":"b5d66550-e67b-4753-9305-078172a6c5b5","Type":"ContainerStarted","Data":"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"}
Sep 08 18:46:22.852209 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:22.851717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" event={"ID":"b5d66550-e67b-4753-9305-078172a6c5b5","Type":"ContainerStarted","Data":"f550899c99a0cc3ac7d3328bae96ba643e3f4a4e77503bbcb111b82dd2e17fce"}
Sep 08 18:46:22.873764 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:22.872532    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" podStartSLOduration=1.8725128020000001 podStartE2EDuration="1.872512802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:22.870448811 +0000 UTC m=+1357.322023944" watchObservedRunningTime="2026-09-08 18:46:22.872512802 +0000 UTC m=+1357.324087936"
Sep 08 18:46:29.733740 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.733692    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:46:29.737259 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.737233    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.740074 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.740043    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ff349c\""
Sep 08 18:46:29.740177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.740123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ff349c-dockercfg-mj7rp\""
Sep 08 18:46:29.757059 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.757030    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:46:29.832622 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.832574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfdw\" (UniqueName: \"kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.832783 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.832636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.934145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.934108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfdw\" (UniqueName: \"kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.934145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.934152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.934880 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.934858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:29.943216 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:29.943191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfdw\" (UniqueName: \"kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw\") pod \"payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:30.050806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.050684    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:30.327095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.327066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:46:30.328741 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:46:30.328697    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c392cad_20a7_40b2_a513_0bf83dc9132a.slice/crio-c3eb1f94d8771e9be46ac2f007eacb79fca816894d179f9bc299191dcd040981 WatchSource:0}: Error finding container c3eb1f94d8771e9be46ac2f007eacb79fca816894d179f9bc299191dcd040981: Status 404 returned error can't find the container with id c3eb1f94d8771e9be46ac2f007eacb79fca816894d179f9bc299191dcd040981
Sep 08 18:46:30.895811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.895766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" event={"ID":"1c392cad-20a7-40b2-a513-0bf83dc9132a","Type":"ContainerStarted","Data":"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"}
Sep 08 18:46:30.896226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.895835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" event={"ID":"1c392cad-20a7-40b2-a513-0bf83dc9132a","Type":"ContainerStarted","Data":"c3eb1f94d8771e9be46ac2f007eacb79fca816894d179f9bc299191dcd040981"}
Sep 08 18:46:30.896226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.895862    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:30.916118 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:30.916064    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" podStartSLOduration=1.9160491020000001 podStartE2EDuration="1.916049102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:30.914371806 +0000 UTC m=+1365.365946948" watchObservedRunningTime="2026-09-08 18:46:30.916049102 +0000 UTC m=+1365.367624234"
Sep 08 18:46:32.029924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:32.029885    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:32.030325 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:32.029942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:32.042858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:32.042833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:32.918377 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:32.918348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:46:41.904366 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:41.904333    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:46:53.568648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.568605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:46:53.574402 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.574366    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.577104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.577081    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-179a6f-openshift-default-dockercfg-bc2g6\""
Sep 08 18:46:53.585933 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.585906    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:46:53.666502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.666462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667051 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667303 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667285    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667980 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.667980 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.667911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtv5\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtv5\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.768876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768939    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.768993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.769171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.769193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.769261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.769378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.769560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.769535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.771394 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.771371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.771552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.771529    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.777272 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.777248    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.777402 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.777382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtv5\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5\") pod \"e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") " pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:53.888161 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:53.888129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:54.166562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:54.166533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:46:54.168569 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:46:54.168538    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebfdcd0_3497_457d_9119_dec8032d8f3c.slice/crio-974af4334a836fe0ff738e9564cda7c5f5e9271ec4ce86759e83435332ab7f60 WatchSource:0}: Error finding container 974af4334a836fe0ff738e9564cda7c5f5e9271ec4ce86759e83435332ab7f60: Status 404 returned error can't find the container with id 974af4334a836fe0ff738e9564cda7c5f5e9271ec4ce86759e83435332ab7f60
Sep 08 18:46:54.170815 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:54.170783    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:46:54.170881 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:54.170852    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:46:54.170923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:54.170885    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:46:55.025923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:55.025883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerStarted","Data":"bfad37e3356728d56aa447e7c2ed852af26bb29c0c2096582c37b96d0d76b670"}
Sep 08 18:46:55.026326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:55.025943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerStarted","Data":"974af4334a836fe0ff738e9564cda7c5f5e9271ec4ce86759e83435332ab7f60"}
Sep 08 18:46:55.047348 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:55.047297    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podStartSLOduration=2.047283028 podStartE2EDuration="2.047283028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:46:55.046078662 +0000 UTC m=+1389.497653999" watchObservedRunningTime="2026-09-08 18:46:55.047283028 +0000 UTC m=+1389.498858160"
Sep 08 18:46:55.888685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:55.888643    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:55.894741 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:55.894700    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:56.031308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:56.031275    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:46:56.032346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:46:56.032324    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:47:20.158885 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.158796    2582 generic.go:356] "Generic (PLEG): container finished" podID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerID="bfad37e3356728d56aa447e7c2ed852af26bb29c0c2096582c37b96d0d76b670" exitCode=0
Sep 08 18:47:20.158885 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.158873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerDied","Data":"bfad37e3356728d56aa447e7c2ed852af26bb29c0c2096582c37b96d0d76b670"}
Sep 08 18:47:20.159335 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.159316    2582 scope.go:117] "RemoveContainer" containerID="bfad37e3356728d56aa447e7c2ed852af26bb29c0c2096582c37b96d0d76b670"
Sep 08 18:47:20.160487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.160443    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:47:20.160590 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.160545    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:47:20.160664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:20.160625    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:47:21.172072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:21.172028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerStarted","Data":"d13765022a677a32f43465299a747b4bfb9ba7edf424ec5bc77302f51bb026c3"}
Sep 08 18:47:21.889041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:21.889001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:47:21.890267 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:21.890234    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 08 18:47:21.890395 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:21.890292    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 08 18:47:22.889105 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:22.889069    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 08 18:47:22.889486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:22.889122    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 08 18:47:23.889486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:23.889446    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 08 18:47:23.889930 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:23.889511    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 08 18:47:24.654888 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.654855    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"]
Sep 08 18:47:24.657880 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.657845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:24.660796 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.660767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:47:24.666462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.666433    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"]
Sep 08 18:47:24.794701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.794662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds\" (UID: \"a98874e4-7116-4db6-8472-158242fa88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:24.888957 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.888921    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 08 18:47:24.889124 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.888977    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 08 18:47:24.895447 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.895419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds\" (UID: \"a98874e4-7116-4db6-8472-158242fa88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:24.896055 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.896037    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds\" (UID: \"a98874e4-7116-4db6-8472-158242fa88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:24.977266 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:24.977167    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:25.249702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:25.249678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"]
Sep 08 18:47:25.251050 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:47:25.251024    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98874e4_7116_4db6_8472_158242fa88d2.slice/crio-0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50 WatchSource:0}: Error finding container 0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50: Status 404 returned error can't find the container with id 0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50
Sep 08 18:47:25.889232 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:25.889195    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Sep 08 18:47:25.889418 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:25.889253    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Sep 08 18:47:26.032235 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.032148    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:47:26.206255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.206217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds" event={"ID":"a98874e4-7116-4db6-8472-158242fa88d2","Type":"ContainerStarted","Data":"875293b9442ba5d27973bdc1247c7a5a82b9d9630ee3dc1562b8d8092c8b24fe"}
Sep 08 18:47:26.206421 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.206292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds" event={"ID":"a98874e4-7116-4db6-8472-158242fa88d2","Type":"ContainerStarted","Data":"0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50"}
Sep 08 18:47:26.225011 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.224946    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds" podStartSLOduration=2.22492858 podStartE2EDuration="2.22492858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:26.221637141 +0000 UTC m=+1420.673212274" watchObservedRunningTime="2026-09-08 18:47:26.22492858 +0000 UTC m=+1420.676503720"
Sep 08 18:47:26.893424 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.893390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:47:26.894523 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:26.894499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:47:27.213510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:27.213418    2582 generic.go:356] "Generic (PLEG): container finished" podID="a98874e4-7116-4db6-8472-158242fa88d2" containerID="875293b9442ba5d27973bdc1247c7a5a82b9d9630ee3dc1562b8d8092c8b24fe" exitCode=0
Sep 08 18:47:27.213510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:27.213462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds" event={"ID":"a98874e4-7116-4db6-8472-158242fa88d2","Type":"ContainerDied","Data":"875293b9442ba5d27973bdc1247c7a5a82b9d9630ee3dc1562b8d8092c8b24fe"}
Sep 08 18:47:28.363422 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:28.363398    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:28.430698 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:28.430664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca\") pod \"a98874e4-7116-4db6-8472-158242fa88d2\" (UID: \"a98874e4-7116-4db6-8472-158242fa88d2\") "
Sep 08 18:47:28.431048 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:28.431026    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a98874e4-7116-4db6-8472-158242fa88d2" (UID: "a98874e4-7116-4db6-8472-158242fa88d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:47:28.532095 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:28.532010    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98874e4-7116-4db6-8472-158242fa88d2-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:29.226165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.226132    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"
Sep 08 18:47:29.226357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.226167    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds" event={"ID":"a98874e4-7116-4db6-8472-158242fa88d2","Type":"ContainerDied","Data":"0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50"}
Sep 08 18:47:29.226357 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.226226    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3daa59521ba1ed76437ae2f8ee0664165a067e2abdb49d9e76ef5243e8cc50"
Sep 08 18:47:29.695782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.695750    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-e55590"]
Sep 08 18:47:29.696162 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.696149    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98874e4-7116-4db6-8472-158242fa88d2" containerName="revoke-keys"
Sep 08 18:47:29.696209 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.696163    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98874e4-7116-4db6-8472-158242fa88d2" containerName="revoke-keys"
Sep 08 18:47:29.696241 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.696225    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98874e4-7116-4db6-8472-158242fa88d2" containerName="revoke-keys"
Sep 08 18:47:29.698407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.698390    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:29.701310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.701289    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mhbkz\""
Sep 08 18:47:29.705060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.705036    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-e55590"]
Sep 08 18:47:29.844805 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.844750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dzl\" (UniqueName: \"kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl\") pod \"test-curl-7578-e55590\" (UID: \"2edda279-34fe-43ae-b35f-8ed89dfdab35\") " pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:29.946226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.946116    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dzl\" (UniqueName: \"kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl\") pod \"test-curl-7578-e55590\" (UID: \"2edda279-34fe-43ae-b35f-8ed89dfdab35\") " pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:29.954504 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:29.954478    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dzl\" (UniqueName: \"kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl\") pod \"test-curl-7578-e55590\" (UID: \"2edda279-34fe-43ae-b35f-8ed89dfdab35\") " pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:30.008597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:30.008559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:30.278319 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:30.278289    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-e55590"]
Sep 08 18:47:30.279660 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:47:30.279637    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2edda279_34fe_43ae_b35f_8ed89dfdab35.slice/crio-0c786b231915729c372fc3ecf00d14e1fe5596a4f2f1ac4e1f9038fede87758d WatchSource:0}: Error finding container 0c786b231915729c372fc3ecf00d14e1fe5596a4f2f1ac4e1f9038fede87758d: Status 404 returned error can't find the container with id 0c786b231915729c372fc3ecf00d14e1fe5596a4f2f1ac4e1f9038fede87758d
Sep 08 18:47:31.238482 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.238448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-e55590" event={"ID":"2edda279-34fe-43ae-b35f-8ed89dfdab35","Type":"ContainerStarted","Data":"2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5"}
Sep 08 18:47:31.238952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.238492    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-e55590" event={"ID":"2edda279-34fe-43ae-b35f-8ed89dfdab35","Type":"ContainerStarted","Data":"0c786b231915729c372fc3ecf00d14e1fe5596a4f2f1ac4e1f9038fede87758d"}
Sep 08 18:47:31.254209 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.254161    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7578-e55590" podStartSLOduration=2.254145996 podStartE2EDuration="2.254145996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:31.25323094 +0000 UTC m=+1425.704806073" watchObservedRunningTime="2026-09-08 18:47:31.254145996 +0000 UTC m=+1425.705721127"
Sep 08 18:47:31.855369 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.855325    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-e55590"]
Sep 08 18:47:31.859084 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.859055    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-e55590"]
Sep 08 18:47:31.971499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.971464    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7578-47c5b6"]
Sep 08 18:47:31.971945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.971930    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edda279-34fe-43ae-b35f-8ed89dfdab35" containerName="test-curl-7578-e55590"
Sep 08 18:47:31.972002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.971947    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edda279-34fe-43ae-b35f-8ed89dfdab35" containerName="test-curl-7578-e55590"
Sep 08 18:47:31.972044 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.972034    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edda279-34fe-43ae-b35f-8ed89dfdab35" containerName="test-curl-7578-e55590"
Sep 08 18:47:31.974076 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.974062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:31.981230 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:31.981194    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-47c5b6"]
Sep 08 18:47:32.065597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:32.065567    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnww\" (UniqueName: \"kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww\") pod \"test-curl-7578-47c5b6\" (UID: \"ba4d06ed-4eca-4316-9479-63e36f26afae\") " pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:32.166440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:32.166404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnww\" (UniqueName: \"kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww\") pod \"test-curl-7578-47c5b6\" (UID: \"ba4d06ed-4eca-4316-9479-63e36f26afae\") " pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:32.176772 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:32.176743    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnww\" (UniqueName: \"kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww\") pod \"test-curl-7578-47c5b6\" (UID: \"ba4d06ed-4eca-4316-9479-63e36f26afae\") " pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:32.249796 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:32.245942    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7578-e55590" container="test-curl-7578-e55590"
Sep 08 18:47:32.284702 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:32.284669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:32.557777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:32.557753    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7578-47c5b6"]
Sep 08 18:47:32.559193 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:47:32.559164    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba4d06ed_4eca_4316_9479_63e36f26afae.slice/crio-75c33e645b8aa18054a2652311fc80d541cf8521d80d36181f9c9cf00cf3bca4 WatchSource:0}: Error finding container 75c33e645b8aa18054a2652311fc80d541cf8521d80d36181f9c9cf00cf3bca4: Status 404 returned error can't find the container with id 75c33e645b8aa18054a2652311fc80d541cf8521d80d36181f9c9cf00cf3bca4
Sep 08 18:47:33.252828 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.252789    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-47c5b6" event={"ID":"ba4d06ed-4eca-4316-9479-63e36f26afae","Type":"ContainerStarted","Data":"9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf"}
Sep 08 18:47:33.252828 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.252832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7578-47c5b6" event={"ID":"ba4d06ed-4eca-4316-9479-63e36f26afae","Type":"ContainerStarted","Data":"75c33e645b8aa18054a2652311fc80d541cf8521d80d36181f9c9cf00cf3bca4"}
Sep 08 18:47:33.253325 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.252920    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7578-e55590" podUID="2edda279-34fe-43ae-b35f-8ed89dfdab35" containerName="test-curl-7578-e55590" containerID="cri-o://2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5" gracePeriod=2
Sep 08 18:47:33.268686 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.268636    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7578-47c5b6" podStartSLOduration=2.268620681 podStartE2EDuration="2.268620681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:33.266463984 +0000 UTC m=+1427.718039117" watchObservedRunningTime="2026-09-08 18:47:33.268620681 +0000 UTC m=+1427.720195793"
Sep 08 18:47:33.843035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.842994    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7578-47c5b6"]
Sep 08 18:47:33.847017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:33.846992    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7578-47c5b6"]
Sep 08 18:47:34.258363 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:34.258334    2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7578-47c5b6" container="test-curl-7578-47c5b6"
Sep 08 18:47:35.265911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.265875    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7578-47c5b6" podUID="ba4d06ed-4eca-4316-9479-63e36f26afae" containerName="test-curl-7578-47c5b6" containerID="cri-o://9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf" gracePeriod=2
Sep 08 18:47:35.400452 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.400430    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:35.499290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.499250    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8dzl\" (UniqueName: \"kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl\") pod \"2edda279-34fe-43ae-b35f-8ed89dfdab35\" (UID: \"2edda279-34fe-43ae-b35f-8ed89dfdab35\") "
Sep 08 18:47:35.501519 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.501488    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl" (OuterVolumeSpecName: "kube-api-access-f8dzl") pod "2edda279-34fe-43ae-b35f-8ed89dfdab35" (UID: "2edda279-34fe-43ae-b35f-8ed89dfdab35"). InnerVolumeSpecName "kube-api-access-f8dzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:35.600350 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.600304    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dzl\" (UniqueName: \"kubernetes.io/projected/2edda279-34fe-43ae-b35f-8ed89dfdab35-kube-api-access-f8dzl\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:35.903763 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:35.903713    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:47:36.101250 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.101220    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edda279-34fe-43ae-b35f-8ed89dfdab35" path="/var/lib/kubelet/pods/2edda279-34fe-43ae-b35f-8ed89dfdab35/volumes"
Sep 08 18:47:36.278528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.278437    2582 generic.go:356] "Generic (PLEG): container finished" podID="2edda279-34fe-43ae-b35f-8ed89dfdab35" containerID="2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5" exitCode=137
Sep 08 18:47:36.278528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.278483    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-e55590"
Sep 08 18:47:36.279001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.278528    2582 scope.go:117] "RemoveContainer" containerID="2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5"
Sep 08 18:47:36.288001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.287984    2582 scope.go:117] "RemoveContainer" containerID="2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5"
Sep 08 18:47:36.288247 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:36.288229    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5\": container with ID starting with 2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5 not found: ID does not exist" containerID="2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5"
Sep 08 18:47:36.288314 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.288254    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5"} err="failed to get container status \"2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5\": rpc error: code = NotFound desc = could not find container \"2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5\": container with ID starting with 2baf983abc4fb99bed4f085e8d5cc16ab5ef3995cc01938001018d9cfcc316c5 not found: ID does not exist"
Sep 08 18:47:36.797841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:36.797808    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:47:37.445298 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:37.445274    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:37.622820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:37.622709    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmnww\" (UniqueName: \"kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww\") pod \"ba4d06ed-4eca-4316-9479-63e36f26afae\" (UID: \"ba4d06ed-4eca-4316-9479-63e36f26afae\") "
Sep 08 18:47:37.624979 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:37.624946    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww" (OuterVolumeSpecName: "kube-api-access-zmnww") pod "ba4d06ed-4eca-4316-9479-63e36f26afae" (UID: "ba4d06ed-4eca-4316-9479-63e36f26afae"). InnerVolumeSpecName "kube-api-access-zmnww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:37.723488 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:37.723452    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmnww\" (UniqueName: \"kubernetes.io/projected/ba4d06ed-4eca-4316-9479-63e36f26afae-kube-api-access-zmnww\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:38.099815 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.099776    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba4d06ed-4eca-4316-9479-63e36f26afae" path="/var/lib/kubelet/pods/ba4d06ed-4eca-4316-9479-63e36f26afae/volumes"
Sep 08 18:47:38.292402 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.292368    2582 generic.go:356] "Generic (PLEG): container finished" podID="ba4d06ed-4eca-4316-9479-63e36f26afae" containerID="9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf" exitCode=137
Sep 08 18:47:38.292402 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.292405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7578-47c5b6"
Sep 08 18:47:38.292620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.292435    2582 scope.go:117] "RemoveContainer" containerID="9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf"
Sep 08 18:47:38.304244 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.304224    2582 scope.go:117] "RemoveContainer" containerID="9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf"
Sep 08 18:47:38.304545 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:38.304525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf\": container with ID starting with 9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf not found: ID does not exist" containerID="9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf"
Sep 08 18:47:38.304628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:38.304554    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf"} err="failed to get container status \"9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf\": rpc error: code = NotFound desc = could not find container \"9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf\": container with ID starting with 9a8f264823797d26a7e0867cefc9c03b9dda51e1430be0b7c9ef7efc928a2bdf not found: ID does not exist"
Sep 08 18:47:40.953897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:40.953858    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:47:40.954632 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:40.954594    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" podUID="1c392cad-20a7-40b2-a513-0bf83dc9132a" containerName="payload-pre-processing" containerID="cri-o://c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc" gracePeriod=30
Sep 08 18:47:41.095968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.095934    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"]
Sep 08 18:47:41.101068 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.101036    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ff349c-qc5ds"]
Sep 08 18:47:41.233691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.233664    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:47:41.311148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.311114    2582 generic.go:356] "Generic (PLEG): container finished" podID="1c392cad-20a7-40b2-a513-0bf83dc9132a" containerID="c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc" exitCode=0
Sep 08 18:47:41.311346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.311160    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"
Sep 08 18:47:41.311346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.311213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" event={"ID":"1c392cad-20a7-40b2-a513-0bf83dc9132a","Type":"ContainerDied","Data":"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"}
Sep 08 18:47:41.311346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.311273    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf" event={"ID":"1c392cad-20a7-40b2-a513-0bf83dc9132a","Type":"ContainerDied","Data":"c3eb1f94d8771e9be46ac2f007eacb79fca816894d179f9bc299191dcd040981"}
Sep 08 18:47:41.311346 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.311304    2582 scope.go:117] "RemoveContainer" containerID="c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"
Sep 08 18:47:41.320069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.320054    2582 scope.go:117] "RemoveContainer" containerID="c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"
Sep 08 18:47:41.320325 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:41.320309    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc\": container with ID starting with c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc not found: ID does not exist" containerID="c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"
Sep 08 18:47:41.320363 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.320333    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc"} err="failed to get container status \"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc\": rpc error: code = NotFound desc = could not find container \"c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc\": container with ID starting with c4fd98c6c7715d789979d33301af90155f904476fb44684b0667bb2018df24cc not found: ID does not exist"
Sep 08 18:47:41.358843 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.358810    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume\") pod \"1c392cad-20a7-40b2-a513-0bf83dc9132a\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") "
Sep 08 18:47:41.358996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.358855    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sfdw\" (UniqueName: \"kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw\") pod \"1c392cad-20a7-40b2-a513-0bf83dc9132a\" (UID: \"1c392cad-20a7-40b2-a513-0bf83dc9132a\") "
Sep 08 18:47:41.359187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.359163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c392cad-20a7-40b2-a513-0bf83dc9132a" (UID: "1c392cad-20a7-40b2-a513-0bf83dc9132a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:47:41.361120 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.361093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw" (OuterVolumeSpecName: "kube-api-access-9sfdw") pod "1c392cad-20a7-40b2-a513-0bf83dc9132a" (UID: "1c392cad-20a7-40b2-a513-0bf83dc9132a"). InnerVolumeSpecName "kube-api-access-9sfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:41.460552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.460511    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c392cad-20a7-40b2-a513-0bf83dc9132a-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.460552 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.460545    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfdw\" (UniqueName: \"kubernetes.io/projected/1c392cad-20a7-40b2-a513-0bf83dc9132a-kube-api-access-9sfdw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:41.636788 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.636755    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:47:41.640765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:41.640736    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ff349c-7c95bbd97d-5qvzf"]
Sep 08 18:47:42.099554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:42.099516    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c392cad-20a7-40b2-a513-0bf83dc9132a" path="/var/lib/kubelet/pods/1c392cad-20a7-40b2-a513-0bf83dc9132a/volumes"
Sep 08 18:47:42.099923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:42.099877    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98874e4-7116-4db6-8472-158242fa88d2" path="/var/lib/kubelet/pods/a98874e4-7116-4db6-8472-158242fa88d2/volumes"
Sep 08 18:47:43.814688 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.814651    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:47:43.815075 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4d06ed-4eca-4316-9479-63e36f26afae" containerName="test-curl-7578-47c5b6"
Sep 08 18:47:43.815075 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815070    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4d06ed-4eca-4316-9479-63e36f26afae" containerName="test-curl-7578-47c5b6"
Sep 08 18:47:43.815151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815082    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c392cad-20a7-40b2-a513-0bf83dc9132a" containerName="payload-pre-processing"
Sep 08 18:47:43.815151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815087    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c392cad-20a7-40b2-a513-0bf83dc9132a" containerName="payload-pre-processing"
Sep 08 18:47:43.815212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815161    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c392cad-20a7-40b2-a513-0bf83dc9132a" containerName="payload-pre-processing"
Sep 08 18:47:43.815212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.815209    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4d06ed-4eca-4316-9479-63e36f26afae" containerName="test-curl-7578-47c5b6"
Sep 08 18:47:43.817968 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.817950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.820953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.820935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-792c35\"/\"rate-test-model-792c35-kserve-self-signed-certs\""
Sep 08 18:47:43.828189 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.828168    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:47:43.985106 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.985282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.985282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.985282 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.985497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:43.985497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:43.985424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgz2b\" (UniqueName: \"kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz2b\" (UniqueName: \"kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086825 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086800    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.086931 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.086917    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.088917 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.088895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.089300 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.089286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.096988 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.096952    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz2b\" (UniqueName: \"kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b\") pod \"rate-test-model-792c35-kserve-7bc549fb8c-ffbwz\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") " pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.127495 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.127468    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:44.401119 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:44.401090    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:47:44.403373 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:47:44.403341    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f81d74c_f3ff_42dc_b9f9_24a5c1bc3c2a.slice/crio-5d271c316057071916b4a00dfd143bde4c4c4122541f548ebec16c208a630524 WatchSource:0}: Error finding container 5d271c316057071916b4a00dfd143bde4c4c4122541f548ebec16c208a630524: Status 404 returned error can't find the container with id 5d271c316057071916b4a00dfd143bde4c4c4122541f548ebec16c208a630524
Sep 08 18:47:45.334461 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:45.334422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" event={"ID":"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a","Type":"ContainerStarted","Data":"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"}
Sep 08 18:47:45.334461 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:45.334474    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" event={"ID":"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a","Type":"ContainerStarted","Data":"5d271c316057071916b4a00dfd143bde4c4c4122541f548ebec16c208a630524"}
Sep 08 18:47:45.352827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:45.352778    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" podStartSLOduration=2.352761315 podStartE2EDuration="2.352761315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:47:45.351307154 +0000 UTC m=+1439.802882286" watchObservedRunningTime="2026-09-08 18:47:45.352761315 +0000 UTC m=+1439.804336479"
Sep 08 18:47:46.284765 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:46.284714    2582 scope.go:117] "RemoveContainer" containerID="df39ac62477785f79d423911b635877a084479d2ec1e73ca3c6bbcbf1d45ab01"
Sep 08 18:47:46.293294 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:46.293272    2582 scope.go:117] "RemoveContainer" containerID="b2453e5a9821f26b64c86b42226a702f15bf7b2bff9f03828e3ffdcfc0ef7ff8"
Sep 08 18:47:46.301194 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:46.301176    2582 scope.go:117] "RemoveContainer" containerID="c5f3ba3343879670537e18ba6c32722b5089fbfc1a2f390f3e0e1d70ae57904e"
Sep 08 18:47:47.808830 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.808795    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"]
Sep 08 18:47:47.812205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.812186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:47.815032 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.815006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:47:47.819476 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.819449    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"]
Sep 08 18:47:47.820393 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.820134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z\" (UID: \"afb6fb18-829b-4798-9a9d-61f2780f5255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:47.920924 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.920883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z\" (UID: \"afb6fb18-829b-4798-9a9d-61f2780f5255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:47.921531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:47.921511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z\" (UID: \"afb6fb18-829b-4798-9a9d-61f2780f5255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:48.124554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:48.124308    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:48.398556 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:48.398530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"]
Sep 08 18:47:48.400790 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:47:48.400762    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb6fb18_829b_4798_9a9d_61f2780f5255.slice/crio-70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303 WatchSource:0}: Error finding container 70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303: Status 404 returned error can't find the container with id 70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303
Sep 08 18:47:49.363104 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:49.363007    2582 generic.go:356] "Generic (PLEG): container finished" podID="afb6fb18-829b-4798-9a9d-61f2780f5255" containerID="c46a9595445e395567e04521ef1a618a023dd3f02bc3bc6598bc8b0a23120da5" exitCode=0
Sep 08 18:47:49.363517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:49.363101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z" event={"ID":"afb6fb18-829b-4798-9a9d-61f2780f5255","Type":"ContainerDied","Data":"c46a9595445e395567e04521ef1a618a023dd3f02bc3bc6598bc8b0a23120da5"}
Sep 08 18:47:49.363517 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:49.363169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z" event={"ID":"afb6fb18-829b-4798-9a9d-61f2780f5255","Type":"ContainerStarted","Data":"70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303"}
Sep 08 18:47:50.506273 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:50.506244    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:50.544415 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:50.544381    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca\") pod \"afb6fb18-829b-4798-9a9d-61f2780f5255\" (UID: \"afb6fb18-829b-4798-9a9d-61f2780f5255\") "
Sep 08 18:47:50.544768 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:50.544743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afb6fb18-829b-4798-9a9d-61f2780f5255" (UID: "afb6fb18-829b-4798-9a9d-61f2780f5255"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:47:50.645544 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:50.645503    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb6fb18-829b-4798-9a9d-61f2780f5255-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:50.924228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:50.924116    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" podUID="32decf86-6654-497f-b5ac-1fa579466a1d" containerName="main" containerID="cri-o://b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20" gracePeriod=45
Sep 08 18:47:51.177891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.177836    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:47:51.251310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2k88\" (UniqueName: \"kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251476    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache\") pod \"32decf86-6654-497f-b5ac-1fa579466a1d\" (UID: \"32decf86-6654-497f-b5ac-1fa579466a1d\") "
Sep 08 18:47:51.251808 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251715    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home" (OuterVolumeSpecName: "home") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:51.251808 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:51.251937 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.251868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:51.253652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.253624    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm" (OuterVolumeSpecName: "dshm") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:51.253652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.253634    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:47:51.253864 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.253654    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88" (OuterVolumeSpecName: "kube-api-access-d2k88") pod "32decf86-6654-497f-b5ac-1fa579466a1d" (UID: "32decf86-6654-497f-b5ac-1fa579466a1d"). InnerVolumeSpecName "kube-api-access-d2k88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:51.352827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352793    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.352827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352822    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.352827 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352832    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2k88\" (UniqueName: \"kubernetes.io/projected/32decf86-6654-497f-b5ac-1fa579466a1d-kube-api-access-d2k88\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.353054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352841    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32decf86-6654-497f-b5ac-1fa579466a1d-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.353054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352850    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.353054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.352858    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32decf86-6654-497f-b5ac-1fa579466a1d-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:51.376351 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.376319    2582 generic.go:356] "Generic (PLEG): container finished" podID="32decf86-6654-497f-b5ac-1fa579466a1d" containerID="b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20" exitCode=0
Sep 08 18:47:51.376499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.376368    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"
Sep 08 18:47:51.376499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.376397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" event={"ID":"32decf86-6654-497f-b5ac-1fa579466a1d","Type":"ContainerDied","Data":"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"}
Sep 08 18:47:51.376499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.376440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf" event={"ID":"32decf86-6654-497f-b5ac-1fa579466a1d","Type":"ContainerDied","Data":"4bdecd3eaefafa3f89babb9739943188ad5feed5769ae74ebd457842d56255b2"}
Sep 08 18:47:51.376499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.376462    2582 scope.go:117] "RemoveContainer" containerID="b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"
Sep 08 18:47:51.377845 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.377791    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"
Sep 08 18:47:51.377845 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.377795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z" event={"ID":"afb6fb18-829b-4798-9a9d-61f2780f5255","Type":"ContainerDied","Data":"70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303"}
Sep 08 18:47:51.377845 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.377837    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70be9f3cfbc1209f47a4d528f8112dc64bf8ed293f32efa6be9bfef0ad803303"
Sep 08 18:47:51.386958 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.386936    2582 scope.go:117] "RemoveContainer" containerID="b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"
Sep 08 18:47:51.387267 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:51.387246    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20\": container with ID starting with b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20 not found: ID does not exist" containerID="b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"
Sep 08 18:47:51.387342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.387278    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20"} err="failed to get container status \"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20\": rpc error: code = NotFound desc = could not find container \"b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20\": container with ID starting with b71012b83efd3a48c2debfb066b41fa9bb37d73b52acc3d89754ff36a4ffdb20 not found: ID does not exist"
Sep 08 18:47:51.399531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.399508    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:47:51.403640 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.403618    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/sub-test-model-792c35-kserve-6494d84bf-wbvvf"]
Sep 08 18:47:51.818825 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:51.818747    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" podUID="b5d66550-e67b-4753-9305-078172a6c5b5" containerName="main" containerID="cri-o://1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b" gracePeriod=45
Sep 08 18:47:52.080375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.080318    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:47:52.100327 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.100276    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32decf86-6654-497f-b5ac-1fa579466a1d" path="/var/lib/kubelet/pods/32decf86-6654-497f-b5ac-1fa579466a1d/volumes"
Sep 08 18:47:52.160774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.160952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.160952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.160952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lggsh\" (UniqueName: \"kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.160952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.161178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:52.161178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.160982    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs\") pod \"b5d66550-e67b-4753-9305-078172a6c5b5\" (UID: \"b5d66550-e67b-4753-9305-078172a6c5b5\") "
Sep 08 18:47:52.161178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.161108    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:52.161341 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.161193    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home" (OuterVolumeSpecName: "home") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:52.161428 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.161410    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.161494 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.161434    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.161494 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.161447    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.163153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.163125    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm" (OuterVolumeSpecName: "dshm") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:47:52.163255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.163195    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh" (OuterVolumeSpecName: "kube-api-access-lggsh") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "kube-api-access-lggsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:47:52.163255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.163197    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5d66550-e67b-4753-9305-078172a6c5b5" (UID: "b5d66550-e67b-4753-9305-078172a6c5b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:47:52.262179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.262142    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d66550-e67b-4753-9305-078172a6c5b5-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.262179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.262171    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d66550-e67b-4753-9305-078172a6c5b5-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.262179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.262184    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lggsh\" (UniqueName: \"kubernetes.io/projected/b5d66550-e67b-4753-9305-078172a6c5b5-kube-api-access-lggsh\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:47:52.384870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.384773    2582 generic.go:356] "Generic (PLEG): container finished" podID="b5d66550-e67b-4753-9305-078172a6c5b5" containerID="1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b" exitCode=0
Sep 08 18:47:52.384870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.384825    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"
Sep 08 18:47:52.385098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.384831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" event={"ID":"b5d66550-e67b-4753-9305-078172a6c5b5","Type":"ContainerDied","Data":"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"}
Sep 08 18:47:52.385098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.384943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn" event={"ID":"b5d66550-e67b-4753-9305-078172a6c5b5","Type":"ContainerDied","Data":"f550899c99a0cc3ac7d3328bae96ba643e3f4a4e77503bbcb111b82dd2e17fce"}
Sep 08 18:47:52.385098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.384967    2582 scope.go:117] "RemoveContainer" containerID="1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"
Sep 08 18:47:52.394360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.394343    2582 scope.go:117] "RemoveContainer" containerID="1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"
Sep 08 18:47:52.394650 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:47:52.394623    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b\": container with ID starting with 1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b not found: ID does not exist" containerID="1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"
Sep 08 18:47:52.394707 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.394650    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b"} err="failed to get container status \"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b\": rpc error: code = NotFound desc = could not find container \"1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b\": container with ID starting with 1c8a2ce54c11cc57c7768e19afefd8e788a733d9cffe2a4cb068ef23b348739b not found: ID does not exist"
Sep 08 18:47:52.407025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.407001    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:47:52.410397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:52.410374    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/sub-test-model-6255f2-kserve-699489bb4c-749zn"]
Sep 08 18:47:54.099576 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:54.099532    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5d66550-e67b-4753-9305-078172a6c5b5" path="/var/lib/kubelet/pods/b5d66550-e67b-4753-9305-078172a6c5b5/volumes"
Sep 08 18:47:54.128445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:54.128416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:54.128613 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:54.128468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:54.141348 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:54.141320    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:47:54.410586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:47:54.410559    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:48:03.802147 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:03.802107    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"]
Sep 08 18:48:03.806570 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:03.806544    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-179a6f-qzn2z"]
Sep 08 18:48:04.099256 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:04.099157    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb6fb18-829b-4798-9a9d-61f2780f5255" path="/var/lib/kubelet/pods/afb6fb18-829b-4798-9a9d-61f2780f5255/volumes"
Sep 08 18:48:10.314449 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.314411    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:48:10.314945 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.314668    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" containerID="cri-o://d13765022a677a32f43465299a747b4bfb9ba7edf424ec5bc77302f51bb026c3" gracePeriod=30
Sep 08 18:48:10.660291 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660252    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"]
Sep 08 18:48:10.660663 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660649    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5d66550-e67b-4753-9305-078172a6c5b5" containerName="main"
Sep 08 18:48:10.660711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660664    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5d66550-e67b-4753-9305-078172a6c5b5" containerName="main"
Sep 08 18:48:10.660711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660674    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32decf86-6654-497f-b5ac-1fa579466a1d" containerName="main"
Sep 08 18:48:10.660711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660679    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32decf86-6654-497f-b5ac-1fa579466a1d" containerName="main"
Sep 08 18:48:10.660711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb6fb18-829b-4798-9a9d-61f2780f5255" containerName="revoke-keys"
Sep 08 18:48:10.660711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660696    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb6fb18-829b-4798-9a9d-61f2780f5255" containerName="revoke-keys"
Sep 08 18:48:10.660906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660781    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5d66550-e67b-4753-9305-078172a6c5b5" containerName="main"
Sep 08 18:48:10.660906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660790    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb6fb18-829b-4798-9a9d-61f2780f5255" containerName="revoke-keys"
Sep 08 18:48:10.660906 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.660799    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32decf86-6654-497f-b5ac-1fa579466a1d" containerName="main"
Sep 08 18:48:10.663759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.663742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:10.666374 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.666354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:48:10.671154 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.671125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"]
Sep 08 18:48:10.745609 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.745574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6\" (UID: \"baf07ec9-2afb-4319-9cdb-cd6e4064a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:10.847033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.846996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6\" (UID: \"baf07ec9-2afb-4319-9cdb-cd6e4064a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:10.847650 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.847630    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6\" (UID: \"baf07ec9-2afb-4319-9cdb-cd6e4064a94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:10.976809 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:10.976705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:11.247159 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.247129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"]
Sep 08 18:48:11.248697 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:48:11.248667    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf07ec9_2afb_4319_9cdb_cd6e4064a94d.slice/crio-89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76 WatchSource:0}: Error finding container 89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76: Status 404 returned error can't find the container with id 89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76
Sep 08 18:48:11.490432 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.490403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6" event={"ID":"baf07ec9-2afb-4319-9cdb-cd6e4064a94d","Type":"ContainerStarted","Data":"57e34dd3ffd2eb52cf7086d2570ebe4cd19603b7b36902c26b41cf134aaf34a5"}
Sep 08 18:48:11.490774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.490452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6" event={"ID":"baf07ec9-2afb-4319-9cdb-cd6e4064a94d","Type":"ContainerStarted","Data":"89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76"}
Sep 08 18:48:11.505375 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.505295    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6" podStartSLOduration=1.5052798699999999 podStartE2EDuration="1.50527987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:11.504030615 +0000 UTC m=+1465.955605750" watchObservedRunningTime="2026-09-08 18:48:11.50527987 +0000 UTC m=+1465.956855001"
Sep 08 18:48:11.895857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.895827    2582 patch_prober.go:28] interesting pod/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 18:48:11.895962 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:11.895885    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:48:12.498199 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:12.498162    2582 generic.go:356] "Generic (PLEG): container finished" podID="baf07ec9-2afb-4319-9cdb-cd6e4064a94d" containerID="57e34dd3ffd2eb52cf7086d2570ebe4cd19603b7b36902c26b41cf134aaf34a5" exitCode=0
Sep 08 18:48:12.498589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:12.498218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6" event={"ID":"baf07ec9-2afb-4319-9cdb-cd6e4064a94d","Type":"ContainerDied","Data":"57e34dd3ffd2eb52cf7086d2570ebe4cd19603b7b36902c26b41cf134aaf34a5"}
Sep 08 18:48:13.646643 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:13.646614    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:13.775804 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:13.775692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca\") pod \"baf07ec9-2afb-4319-9cdb-cd6e4064a94d\" (UID: \"baf07ec9-2afb-4319-9cdb-cd6e4064a94d\") "
Sep 08 18:48:13.776030 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:13.776008    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "baf07ec9-2afb-4319-9cdb-cd6e4064a94d" (UID: "baf07ec9-2afb-4319-9cdb-cd6e4064a94d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:13.876490 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:13.876448    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baf07ec9-2afb-4319-9cdb-cd6e4064a94d-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:14.511514 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:14.511481    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"
Sep 08 18:48:14.511677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:14.511482    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6" event={"ID":"baf07ec9-2afb-4319-9cdb-cd6e4064a94d","Type":"ContainerDied","Data":"89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76"}
Sep 08 18:48:14.511677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:14.511602    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89db9e65c1b8cd0473b03eda31bceba78bbed420be29a4a4b577fc954b681a76"
Sep 08 18:48:15.521183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.521153    2582 generic.go:356] "Generic (PLEG): container finished" podID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerID="d13765022a677a32f43465299a747b4bfb9ba7edf424ec5bc77302f51bb026c3" exitCode=0
Sep 08 18:48:15.521574 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.521221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerDied","Data":"d13765022a677a32f43465299a747b4bfb9ba7edf424ec5bc77302f51bb026c3"}
Sep 08 18:48:15.521574 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.521281    2582 scope.go:117] "RemoveContainer" containerID="bfad37e3356728d56aa447e7c2ed852af26bb29c0c2096582c37b96d0d76b670"
Sep 08 18:48:15.577228 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.577170    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:48:15.694088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694054    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694097    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694177    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694253 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694208    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694304    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgtv5\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694327    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694459    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:15.694668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694494    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:15.694668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket\") pod \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\" (UID: \"3ebfdcd0-3497-457d-9119-dec8032d8f3c\") "
Sep 08 18:48:15.694668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694574    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:15.694847 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:15.694902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694842    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:15.695012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.694997    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.695063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.695015    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.695063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.695024    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.695063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.695032    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.695063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.695040    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.697133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.697093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:48:15.697133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.697093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:15.697279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.697172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:15.697279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.697192    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5" (OuterVolumeSpecName: "kube-api-access-cgtv5") pod "3ebfdcd0-3497-457d-9119-dec8032d8f3c" (UID: "3ebfdcd0-3497-457d-9119-dec8032d8f3c"). InnerVolumeSpecName "kube-api-access-cgtv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:15.796185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.796144    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtv5\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-kube-api-access-cgtv5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.796185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.796178    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.796185 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.796189    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:15.796419 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:15.796197    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ebfdcd0-3497-457d-9119-dec8032d8f3c-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:16.527797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:16.527768    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"
Sep 08 18:48:16.528184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:16.527775    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5" event={"ID":"3ebfdcd0-3497-457d-9119-dec8032d8f3c","Type":"ContainerDied","Data":"974af4334a836fe0ff738e9564cda7c5f5e9271ec4ce86759e83435332ab7f60"}
Sep 08 18:48:16.528184 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:16.527898    2582 scope.go:117] "RemoveContainer" containerID="d13765022a677a32f43465299a747b4bfb9ba7edf424ec5bc77302f51bb026c3"
Sep 08 18:48:16.546878 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:16.546853    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:48:16.550589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:16.550562    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-179a6f-openshift-default-746bb7844b-gkfw5"]
Sep 08 18:48:18.100515 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:18.100480    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" path="/var/lib/kubelet/pods/3ebfdcd0-3497-457d-9119-dec8032d8f3c/volumes"
Sep 08 18:48:20.912078 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912033    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912483    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912497    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912505    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912510    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912519    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf07ec9-2afb-4319-9cdb-cd6e4064a94d" containerName="revoke-keys"
Sep 08 18:48:20.912597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912525    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf07ec9-2afb-4319-9cdb-cd6e4064a94d" containerName="revoke-keys"
Sep 08 18:48:20.912821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912613    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf07ec9-2afb-4319-9cdb-cd6e4064a94d" containerName="revoke-keys"
Sep 08 18:48:20.912821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912624    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.912821 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.912633    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebfdcd0-3497-457d-9119-dec8032d8f3c" containerName="istio-proxy"
Sep 08 18:48:20.917195 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.917181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:20.921310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.921289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"openshift-service-ca.crt\""
Sep 08 18:48:20.921435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.921289    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"rate-test-model-6255f2-kserve-self-signed-certs\""
Sep 08 18:48:20.921435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.921288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"default-dockercfg-l4kvl\""
Sep 08 18:48:20.921435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.921289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6255f2\"/\"kube-root-ca.crt\""
Sep 08 18:48:20.925093 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:20.925072    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:48:21.048165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.048165 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.048410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.048410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdrz\" (UniqueName: \"kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.048410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.048410 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.048341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149212 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149386 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149230    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdrz\" (UniqueName: \"kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149386 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.149956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.149946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.150338 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.150316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.152048 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.152007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.152669 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.152648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.161616 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.161423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdrz\" (UniqueName: \"kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz\") pod \"rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") " pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.227561 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.227478    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:21.310746 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.310675    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"]
Sep 08 18:48:21.314487 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.314458    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-3b04d6-k5mb6"]
Sep 08 18:48:21.498538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.498508    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:48:21.499979 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:48:21.499952    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc38f2b2a_0e84_4d86_8b35_eec71dca3277.slice/crio-f91ac3a19b545bb87ff34909a11fe35f612732c41fd63cb0f9885463c8643c6d WatchSource:0}: Error finding container f91ac3a19b545bb87ff34909a11fe35f612732c41fd63cb0f9885463c8643c6d: Status 404 returned error can't find the container with id f91ac3a19b545bb87ff34909a11fe35f612732c41fd63cb0f9885463c8643c6d
Sep 08 18:48:21.557451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:21.557415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" event={"ID":"c38f2b2a-0e84-4d86-8b35-eec71dca3277","Type":"ContainerStarted","Data":"f91ac3a19b545bb87ff34909a11fe35f612732c41fd63cb0f9885463c8643c6d"}
Sep 08 18:48:22.099536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:22.099501    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf07ec9-2afb-4319-9cdb-cd6e4064a94d" path="/var/lib/kubelet/pods/baf07ec9-2afb-4319-9cdb-cd6e4064a94d/volumes"
Sep 08 18:48:22.565120 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:22.565083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" event={"ID":"c38f2b2a-0e84-4d86-8b35-eec71dca3277","Type":"ContainerStarted","Data":"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"}
Sep 08 18:48:22.582258 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:22.582199    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" podStartSLOduration=2.582180555 podStartE2EDuration="2.582180555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:22.581804927 +0000 UTC m=+1477.033380060" watchObservedRunningTime="2026-09-08 18:48:22.582180555 +0000 UTC m=+1477.033755724"
Sep 08 18:48:27.460593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:27.460558    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:48:27.461051 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:27.460861    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy" containerID="cri-o://0db9d1775b92f2494ad9106c2ad02f97715ae3283fd1b6b44f67c3fadfc94a16" gracePeriod=30
Sep 08 18:48:31.227954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:31.227914    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:31.228385 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:31.227972    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:31.240471 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:31.240447    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:31.625040 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:31.624951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:48:32.621340 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.621311    2582 generic.go:356] "Generic (PLEG): container finished" podID="0475295f-9232-4171-ae9a-8965992177da" containerID="0db9d1775b92f2494ad9106c2ad02f97715ae3283fd1b6b44f67c3fadfc94a16" exitCode=0
Sep 08 18:48:32.621807 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.621383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerDied","Data":"0db9d1775b92f2494ad9106c2ad02f97715ae3283fd1b6b44f67c3fadfc94a16"}
Sep 08 18:48:32.621807 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.621445    2582 scope.go:117] "RemoveContainer" containerID="caa94830df8953fa5017e08bcf5db0ea1bb25c5d9a8ee09f4b0b7621d6dfec9e"
Sep 08 18:48:32.712240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.712217    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:48:32.754941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.754864    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.754941 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.754936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.754971    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755003    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zfzh\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755047    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755087    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755146 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755118    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755411 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755156    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755411 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755204    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo\") pod \"0475295f-9232-4171-ae9a-8965992177da\" (UID: \"0475295f-9232-4171-ae9a-8965992177da\") "
Sep 08 18:48:32.755529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:32.755529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755384    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data" (OuterVolumeSpecName: "istio-data") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:32.755529 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755419    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:32.755852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755767    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:32.755852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.755803    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:32.757578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.757548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:32.757707 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.757587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:48:32.757985 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.757958    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token" (OuterVolumeSpecName: "istio-token") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:32.758072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.757994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh" (OuterVolumeSpecName: "kube-api-access-6zfzh") pod "0475295f-9232-4171-ae9a-8965992177da" (UID: "0475295f-9232-4171-ae9a-8965992177da"). InnerVolumeSpecName "kube-api-access-6zfzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:32.856108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856076    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856106    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856115    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0475295f-9232-4171-ae9a-8965992177da-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856108 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856123    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zfzh\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-kube-api-access-6zfzh\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856132    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856141    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856148    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0475295f-9232-4171-ae9a-8965992177da-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856155    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0475295f-9232-4171-ae9a-8965992177da-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:32.856381 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:32.856162    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0475295f-9232-4171-ae9a-8965992177da-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:33.628060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:33.628030    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"
Sep 08 18:48:33.628510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:33.628021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p" event={"ID":"0475295f-9232-4171-ae9a-8965992177da","Type":"ContainerDied","Data":"b11d094eb28fef4f0794485a8cbbef0594ebac4f8b6a0a2842c07698eb96b811"}
Sep 08 18:48:33.628510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:33.628178    2582 scope.go:117] "RemoveContainer" containerID="0db9d1775b92f2494ad9106c2ad02f97715ae3283fd1b6b44f67c3fadfc94a16"
Sep 08 18:48:33.652325 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:33.652300    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:48:33.654138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:33.654119    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-3b04d6-openshift-default-55647b5c6-wsc9p"]
Sep 08 18:48:34.099450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:34.099408    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0475295f-9232-4171-ae9a-8965992177da" path="/var/lib/kubelet/pods/0475295f-9232-4171-ae9a-8965992177da/volumes"
Sep 08 18:48:38.220003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:38.219968    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"]
Sep 08 18:48:38.223066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:38.223042    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-951c3e-29814885-k2bjf"]
Sep 08 18:48:40.098628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:40.098596    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2d7849-87e0-4e63-b87f-d1945a2666f2" path="/var/lib/kubelet/pods/fd2d7849-87e0-4e63-b87f-d1945a2666f2/volumes"
Sep 08 18:48:44.763234 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:44.763201    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:48:44.763707 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:44.763418    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy" containerID="cri-o://d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db" gracePeriod=30
Sep 08 18:48:50.025129 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.025101    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:48:50.114914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsmkv\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.114914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.114914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114896    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114930    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.114990    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115032    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115183 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115115    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket\") pod \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\" (UID: \"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe\") "
Sep 08 18:48:50.115493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:48:50.115493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115299    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:50.115493 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115464    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:50.115634 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115394    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.115634 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115514    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.115634 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115518    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:50.115784 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.115638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:50.117709 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.117681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:48:50.117820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.117765    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv" (OuterVolumeSpecName: "kube-api-access-zsmkv") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "kube-api-access-zsmkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:50.117905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.117883    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:48:50.117964 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.117935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" (UID: "bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:48:50.216456 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216416    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216456 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216452    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216466    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216477    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216489    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216502    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmkv\" (UniqueName: \"kubernetes.io/projected/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-kube-api-access-zsmkv\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.216665 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.216514    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:48:50.726526 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.726485    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerID="d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db" exitCode=0
Sep 08 18:48:50.726676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.726543    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"
Sep 08 18:48:50.726676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.726578    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerDied","Data":"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"}
Sep 08 18:48:50.726676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.726639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh" event={"ID":"bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe","Type":"ContainerDied","Data":"bcc523c17d065f95b7ca1bc1b3babc53979c225cef017fede4d89a34d9192724"}
Sep 08 18:48:50.726676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.726661    2582 scope.go:117] "RemoveContainer" containerID="d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"
Sep 08 18:48:50.736152 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.736132    2582 scope.go:117] "RemoveContainer" containerID="00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"
Sep 08 18:48:50.744510 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.744492    2582 scope.go:117] "RemoveContainer" containerID="d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"
Sep 08 18:48:50.744779 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:48:50.744760    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db\": container with ID starting with d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db not found: ID does not exist" containerID="d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"
Sep 08 18:48:50.744820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.744788    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db"} err="failed to get container status \"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db\": rpc error: code = NotFound desc = could not find container \"d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db\": container with ID starting with d7f29f9e1661a2d900404949bb87bd94017ba30cb23e00d93d9112f4430313db not found: ID does not exist"
Sep 08 18:48:50.744820 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.744803    2582 scope.go:117] "RemoveContainer" containerID="00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"
Sep 08 18:48:50.746018 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:48:50.745864    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413\": container with ID starting with 00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413 not found: ID does not exist" containerID="00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"
Sep 08 18:48:50.746018 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.745896    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413"} err="failed to get container status \"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413\": rpc error: code = NotFound desc = could not find container \"00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413\": container with ID starting with 00c4f72803b7eec9980272236e7023493e9dd091149606a2b94d48876e331413 not found: ID does not exist"
Sep 08 18:48:50.750850 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.750828    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:48:50.752623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:50.752595    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-951c3e-openshift-default-64b955cf9b-bx9bh"]
Sep 08 18:48:52.098830 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:48:52.098794    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" path="/var/lib/kubelet/pods/bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe/volumes"
Sep 08 18:49:02.180002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:02.179969    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:49:03.118863 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:03.118826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:49:17.199901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.199861    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" podUID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" containerName="main" containerID="cri-o://c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1" gracePeriod=45
Sep 08 18:49:17.459701 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.459637    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:49:17.555686 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555654    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.555912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.555912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555715    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.555912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgz2b\" (UniqueName: \"kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.555912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555808    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.555912 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.555852    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache\") pod \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\" (UID: \"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a\") "
Sep 08 18:49:17.556186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.556016    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:17.556186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.556031    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home" (OuterVolumeSpecName: "home") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:17.556186 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.556125    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:17.556307 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.556187    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.556307 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.556203    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.558025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.558001    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm" (OuterVolumeSpecName: "dshm") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:17.558025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.558013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b" (OuterVolumeSpecName: "kube-api-access-wgz2b") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "kube-api-access-wgz2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:49:17.558169 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.558068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" (UID: "9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:49:17.656671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.656635    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.656671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.656667    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgz2b\" (UniqueName: \"kubernetes.io/projected/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-kube-api-access-wgz2b\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.656671 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.656679    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.656937 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.656688    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:17.864562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.864463    2582 generic.go:356] "Generic (PLEG): container finished" podID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" containerID="c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1" exitCode=0
Sep 08 18:49:17.864562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.864508    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"
Sep 08 18:49:17.864797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.864552    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" event={"ID":"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a","Type":"ContainerDied","Data":"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"}
Sep 08 18:49:17.864797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.864611    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz" event={"ID":"9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a","Type":"ContainerDied","Data":"5d271c316057071916b4a00dfd143bde4c4c4122541f548ebec16c208a630524"}
Sep 08 18:49:17.864797 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.864641    2582 scope.go:117] "RemoveContainer" containerID="c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"
Sep 08 18:49:17.874178 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.874156    2582 scope.go:117] "RemoveContainer" containerID="c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"
Sep 08 18:49:17.874458 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:49:17.874439    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1\": container with ID starting with c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1 not found: ID does not exist" containerID="c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"
Sep 08 18:49:17.874499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.874468    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1"} err="failed to get container status \"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1\": rpc error: code = NotFound desc = could not find container \"c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1\": container with ID starting with c5e286074f7ae0f6ebb74af53166594203adc9c6a1dd48e82b8369e2947745d1 not found: ID does not exist"
Sep 08 18:49:17.887670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.887638    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:49:17.893742 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:17.893704    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-792c35/rate-test-model-792c35-kserve-7bc549fb8c-ffbwz"]
Sep 08 18:49:18.099477 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.099444    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" path="/var/lib/kubelet/pods/9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a/volumes"
Sep 08 18:49:18.141470 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.141428    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" podUID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" containerName="main" containerID="cri-o://2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562" gracePeriod=45
Sep 08 18:49:18.396426 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.396350    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:49:18.464594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.464594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.464887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464652    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qdrz\" (UniqueName: \"kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.464887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.464887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.464887 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464718    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache\") pod \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\" (UID: \"c38f2b2a-0e84-4d86-8b35-eec71dca3277\") "
Sep 08 18:49:18.465098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.464969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:18.465098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.465010    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home" (OuterVolumeSpecName: "home") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:18.465098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.465082    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache" (OuterVolumeSpecName: "model-cache") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:18.466970 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.466942    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm" (OuterVolumeSpecName: "dshm") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:49:18.466970 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.466959    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz" (OuterVolumeSpecName: "kube-api-access-8qdrz") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "kube-api-access-8qdrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:49:18.467144 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.466998    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c38f2b2a-0e84-4d86-8b35-eec71dca3277" (UID: "c38f2b2a-0e84-4d86-8b35-eec71dca3277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:49:18.566121 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566089    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.566299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566138    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.566299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566149    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qdrz\" (UniqueName: \"kubernetes.io/projected/c38f2b2a-0e84-4d86-8b35-eec71dca3277-kube-api-access-8qdrz\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.566299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566159    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c38f2b2a-0e84-4d86-8b35-eec71dca3277-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.566299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566167    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.566299 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.566174    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c38f2b2a-0e84-4d86-8b35-eec71dca3277-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:49:18.871684 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.871585    2582 generic.go:356] "Generic (PLEG): container finished" podID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" containerID="2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562" exitCode=0
Sep 08 18:49:18.871684 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.871632    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"
Sep 08 18:49:18.871684 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.871653    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" event={"ID":"c38f2b2a-0e84-4d86-8b35-eec71dca3277","Type":"ContainerDied","Data":"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"}
Sep 08 18:49:18.871983 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.871690    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp" event={"ID":"c38f2b2a-0e84-4d86-8b35-eec71dca3277","Type":"ContainerDied","Data":"f91ac3a19b545bb87ff34909a11fe35f612732c41fd63cb0f9885463c8643c6d"}
Sep 08 18:49:18.871983 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.871707    2582 scope.go:117] "RemoveContainer" containerID="2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"
Sep 08 18:49:18.883135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.883113    2582 scope.go:117] "RemoveContainer" containerID="2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"
Sep 08 18:49:18.884010 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:49:18.883984    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562\": container with ID starting with 2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562 not found: ID does not exist" containerID="2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"
Sep 08 18:49:18.884105 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.884020    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562"} err="failed to get container status \"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562\": rpc error: code = NotFound desc = could not find container \"2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562\": container with ID starting with 2e23ce3339ccc3e5f3b14d173e7603b59b50df0fd618c29fe06031d56376b562 not found: ID does not exist"
Sep 08 18:49:18.894440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.894417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:49:18.896105 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:18.896084    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6255f2/rate-test-model-6255f2-kserve-5bfc68dc5f-rdrvp"]
Sep 08 18:49:20.099539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:20.099505    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" path="/var/lib/kubelet/pods/c38f2b2a-0e84-4d86-8b35-eec71dca3277/volumes"
Sep 08 18:49:32.884996 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.884961    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:49:32.885502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885479    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885497    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885513    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" containerName="main"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885522    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" containerName="main"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885532    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" containerName="main"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885540    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" containerName="main"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885566    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885575    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885585    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885593    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885605    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885613    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885692    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885702    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885710    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f81d74c-f3ff-42dc-b9f9-24a5c1bc3c2a" containerName="main"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885717    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb75f18a-5e46-4c5b-a9e7-0e179a3ea0fe" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885743    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0475295f-9232-4171-ae9a-8965992177da" containerName="istio-proxy"
Sep 08 18:49:32.885891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.885751    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38f2b2a-0e84-4d86-8b35-eec71dca3277" containerName="main"
Sep 08 18:49:32.890585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.890563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.893310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.893287    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-302afb-openshift-default-dockercfg-j2c87\""
Sep 08 18:49:32.900652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.900623    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:49:32.996954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.996920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997235 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997401 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997549 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997549 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997470    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgq9t\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997601    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:32.997664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:32.997626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgq9t\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.098876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.098807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099367 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099479 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.099746 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.099707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.101664 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.101643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.101895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.101877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.106810 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.106790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.106810 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.106805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgq9t\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t\") pod \"e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") " pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.205695 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.205668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:33.480934 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.480909    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:49:33.482719 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:49:33.482685    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f55d1c6_ab0b_4103_b7a1_b0899c9820da.slice/crio-c68c9dc50fec6473baf5706c2ce0e5a3b0d0544ab5b1a746aa06c60615655dfb WatchSource:0}: Error finding container c68c9dc50fec6473baf5706c2ce0e5a3b0d0544ab5b1a746aa06c60615655dfb: Status 404 returned error can't find the container with id c68c9dc50fec6473baf5706c2ce0e5a3b0d0544ab5b1a746aa06c60615655dfb
Sep 08 18:49:33.484850 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.484821    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:33.484918 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.484881    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:33.484918 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.484910    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:33.948704 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.948665    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerStarted","Data":"ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760"}
Sep 08 18:49:33.949182 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.948738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerStarted","Data":"c68c9dc50fec6473baf5706c2ce0e5a3b0d0544ab5b1a746aa06c60615655dfb"}
Sep 08 18:49:33.968331 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:33.968273    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podStartSLOduration=1.968254239 podStartE2EDuration="1.968254239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:33.967686104 +0000 UTC m=+1548.419261257" watchObservedRunningTime="2026-09-08 18:49:33.968254239 +0000 UTC m=+1548.419829373"
Sep 08 18:49:34.206492 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:34.206404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:34.211584 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:34.211560    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:34.953833 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:34.953793    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:34.954667 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:34.954647    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:37.702531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.702494    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:49:37.706135 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.706114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.709440 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.709419    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-302afb-dockercfg-tvnkz\""
Sep 08 18:49:37.709578 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.709560    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-302afb\""
Sep 08 18:49:37.717685 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.717661    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:49:37.846883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.846845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2k5\" (UniqueName: \"kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.847093 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.846907    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.948425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.948384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2k5\" (UniqueName: \"kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.948642 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.948442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.949138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.949118    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:37.956311 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:37.956245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2k5\" (UniqueName: \"kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5\") pod \"payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:38.018320 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.018285    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:38.292699 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.292673    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:49:38.294922 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:49:38.294892    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78359ff6_262c_4b8b_8034_79aa77618678.slice/crio-014bd2094316c07b5ab8acff6626834dd0145dd8f14ff6833d0e710f7b1a7d22 WatchSource:0}: Error finding container 014bd2094316c07b5ab8acff6626834dd0145dd8f14ff6833d0e710f7b1a7d22: Status 404 returned error can't find the container with id 014bd2094316c07b5ab8acff6626834dd0145dd8f14ff6833d0e710f7b1a7d22
Sep 08 18:49:38.976959 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.976921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" event={"ID":"78359ff6-262c-4b8b-8034-79aa77618678","Type":"ContainerStarted","Data":"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"}
Sep 08 18:49:38.977435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.977024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:38.977435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.977049    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" event={"ID":"78359ff6-262c-4b8b-8034-79aa77618678","Type":"ContainerStarted","Data":"014bd2094316c07b5ab8acff6626834dd0145dd8f14ff6833d0e710f7b1a7d22"}
Sep 08 18:49:38.993897 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:38.993850    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" podStartSLOduration=1.993833398 podStartE2EDuration="1.993833398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:38.992302074 +0000 UTC m=+1553.443877206" watchObservedRunningTime="2026-09-08 18:49:38.993833398 +0000 UTC m=+1553.445408560"
Sep 08 18:49:49.985102 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:49.985070    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:49:50.994910 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:50.994864    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 08 18:49:50.995316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:50.994935    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 18:49:53.050317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.050276    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerID="ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760" exitCode=0
Sep 08 18:49:53.050782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.050318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerDied","Data":"ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760"}
Sep 08 18:49:53.050782 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.050698    2582 scope.go:117] "RemoveContainer" containerID="ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760"
Sep 08 18:49:53.051799 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.051772    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:53.051872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.051839    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:53.051872 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:53.051869    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:49:54.058147 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:54.058103    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerStarted","Data":"26b306a7f0605eec48f823c36aa8851ecd86131aefc64893830760839993bf20"}
Sep 08 18:49:54.206799 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:54.206760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:49:54.207029 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:54.207009    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:54.207079 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:54.207053    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:49:55.206217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:55.206177    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:55.206715 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:55.206245    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:49:56.206617 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:56.206576    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:56.207038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:56.206654    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:49:57.206658 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:57.206619    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:57.207106 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:57.206676    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:49:58.206981 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:58.206938    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:58.207387 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:58.207010    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:49:59.206219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:59.206174    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Sep 08 18:49:59.206404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:49:59.206239    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Sep 08 18:50:00.210558 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:00.210524    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:50:00.211035 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:00.210926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:50:00.211717 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:00.211699    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:50:05.932959 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:05.932930    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 18:50:20.214217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.214181    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:50:20.217846 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.217818    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.222568 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.222537    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bc23ef\"/\"kube-root-ca.crt\""
Sep 08 18:50:20.223041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.222861    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bc23ef\"/\"openshift-service-ca.crt\""
Sep 08 18:50:20.223041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.222863    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bc23ef\"/\"ipp-route-bc23ef-kserve-self-signed-certs\""
Sep 08 18:50:20.223041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.222925    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bc23ef\"/\"default-dockercfg-6nz9p\""
Sep 08 18:50:20.226554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.226090    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:50:20.340324 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.340501 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.340501 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6l8t\" (UniqueName: \"kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.340501 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.340501 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.340652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.340554    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.441900 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.441864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.441968    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6l8t\" (UniqueName: \"kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442092 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442327 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442391 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.442445 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.442436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.444478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.444455    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.444811 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.444793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.454862 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.454841    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6l8t\" (UniqueName: \"kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t\") pod \"ipp-route-bc23ef-kserve-64945657c5-44f9b\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") " pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.530390 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.530304    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:20.805286 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:20.805263    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:50:20.807091 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:50:20.807058    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e4360d_011d_491c_bb67_add0765013b4.slice/crio-71a27113024eabde8b50c29cdfaf4c69a001168717f6d983d0d7302caa22f795 WatchSource:0}: Error finding container 71a27113024eabde8b50c29cdfaf4c69a001168717f6d983d0d7302caa22f795: Status 404 returned error can't find the container with id 71a27113024eabde8b50c29cdfaf4c69a001168717f6d983d0d7302caa22f795
Sep 08 18:50:21.215257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:21.215220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" event={"ID":"56e4360d-011d-491c-bb67-add0765013b4","Type":"ContainerStarted","Data":"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"}
Sep 08 18:50:21.215257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:21.215270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" event={"ID":"56e4360d-011d-491c-bb67-add0765013b4","Type":"ContainerStarted","Data":"71a27113024eabde8b50c29cdfaf4c69a001168717f6d983d0d7302caa22f795"}
Sep 08 18:50:21.232127 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:21.232078    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" podStartSLOduration=1.232065057 podStartE2EDuration="1.232065057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:50:21.231649284 +0000 UTC m=+1595.683224419" watchObservedRunningTime="2026-09-08 18:50:21.232065057 +0000 UTC m=+1595.683640190"
Sep 08 18:50:30.531241 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:30.531204    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:30.531645 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:30.531260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:30.543870 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:30.543838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:31.275219 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:31.275190    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:50:52.225139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:52.225052    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 18:50:52.954748 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:52.953832    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:50:52.960565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:52.960536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:52.963857 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:52.963831    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-61f4b7-openshift-default-dockercfg-t4fcj\""
Sep 08 18:50:52.971396 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:52.971375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:50:53.032586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032633    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-452zk\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032686    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.032902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.033243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.032944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.033243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.033009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134421 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134938 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134938 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.134938 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-452zk\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.135120 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.134946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.135181 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.135130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.135647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.135622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.135855 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.135833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.135919 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.135843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.137107 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.137086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.137606 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.137582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.144233 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.144201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.144329 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.144299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-452zk\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk\") pod \"e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") " pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.271927 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.271838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:53.550508 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.550471    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:50:53.551909 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:50:53.551882    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3243def_385c_4827_9d1e_e9b48f928b70.slice/crio-d89f6c976217648ef284993abe482d41d87112074ac3307a85ae6f9d9a73c2c8 WatchSource:0}: Error finding container d89f6c976217648ef284993abe482d41d87112074ac3307a85ae6f9d9a73c2c8: Status 404 returned error can't find the container with id d89f6c976217648ef284993abe482d41d87112074ac3307a85ae6f9d9a73c2c8
Sep 08 18:50:53.554455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.554416    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:50:53.554527 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.554514    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:50:53.554570 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:53.554553    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:50:54.390231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:54.390188    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerStarted","Data":"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"}
Sep 08 18:50:54.390644 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:54.390244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerStarted","Data":"d89f6c976217648ef284993abe482d41d87112074ac3307a85ae6f9d9a73c2c8"}
Sep 08 18:50:54.413856 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:54.413805    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podStartSLOduration=2.41378855 podStartE2EDuration="2.41378855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:50:54.411990499 +0000 UTC m=+1628.863565630" watchObservedRunningTime="2026-09-08 18:50:54.41378855 +0000 UTC m=+1628.865363681"
Sep 08 18:50:55.272585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:55.272551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:55.277697 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:55.277670    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:55.395145 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:55.395114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:55.396486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:55.396468    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:50:58.127769 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.127734    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:50:58.131502 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.131486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.135317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.135291    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-61f4b7\""
Sep 08 18:50:58.135438 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.135325    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-61f4b7-dockercfg-p7lnb\""
Sep 08 18:50:58.146275 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.146248    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:50:58.182451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.182424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.182593 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.182472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5m8\" (UniqueName: \"kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.283802 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.283763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.284001 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.283827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5m8\" (UniqueName: \"kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.284506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.284484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.293454 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.293429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5m8\" (UniqueName: \"kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8\") pod \"payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.442514 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.442482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:58.715435 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:58.715404    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:50:58.715905 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:50:58.715873    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b8fd34_2bd9_4d56_8ae9_902d0d7c596c.slice/crio-13fff722393ef0d8db67ef47de6acc32d174e2ae578b0c42f83023f47a0b8f63 WatchSource:0}: Error finding container 13fff722393ef0d8db67ef47de6acc32d174e2ae578b0c42f83023f47a0b8f63: Status 404 returned error can't find the container with id 13fff722393ef0d8db67ef47de6acc32d174e2ae578b0c42f83023f47a0b8f63
Sep 08 18:50:59.419417 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:59.419380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" event={"ID":"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c","Type":"ContainerStarted","Data":"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"}
Sep 08 18:50:59.419909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:59.419439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" event={"ID":"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c","Type":"ContainerStarted","Data":"13fff722393ef0d8db67ef47de6acc32d174e2ae578b0c42f83023f47a0b8f63"}
Sep 08 18:50:59.419909 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:59.419467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:50:59.438959 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:50:59.438911    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" podStartSLOduration=1.438896586 podStartE2EDuration="1.438896586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:50:59.437211378 +0000 UTC m=+1633.888786512" watchObservedRunningTime="2026-09-08 18:50:59.438896586 +0000 UTC m=+1633.890471719"
Sep 08 18:51:10.428226 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:10.428194    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:51:16.523453 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.523420    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3243def-385c-4827-9d1e-e9b48f928b70" containerID="56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda" exitCode=0
Sep 08 18:51:16.523908 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.523503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerDied","Data":"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"}
Sep 08 18:51:16.524012 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.523997    2582 scope.go:117] "RemoveContainer" containerID="56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"
Sep 08 18:51:16.525148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.525120    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:51:16.525250 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.525202    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:51:16.525314 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:16.525251    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 08 18:51:17.531566 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:17.531528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerStarted","Data":"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"}
Sep 08 18:51:18.273163 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:18.273114    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:18.273499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:18.273212    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 08 18:51:18.273621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:18.273533    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 08 18:51:19.273670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:19.273224    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 08 18:51:19.273670 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:19.273291    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 08 18:51:20.272949 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:20.272899    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 08 18:51:20.273167 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:20.272974    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 08 18:51:21.272506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:21.272464    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 08 18:51:21.273139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:21.273108    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 08 18:51:22.273037 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.272995    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 08 18:51:22.273686 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.273648    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 08 18:51:22.279326 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.279285    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"]
Sep 08 18:51:22.282595 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.282565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:22.287046 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.287028    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:51:22.304647 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.304626    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"]
Sep 08 18:51:22.418864 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.418821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4\" (UID: \"7ed3a561-20ee-4f44-8c71-b92303967f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:22.519942 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.519902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4\" (UID: \"7ed3a561-20ee-4f44-8c71-b92303967f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:22.520524 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.520505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4\" (UID: \"7ed3a561-20ee-4f44-8c71-b92303967f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:22.593127 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.593039    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:22.868407 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:22.868381    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"]
Sep 08 18:51:22.870635 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:51:22.870604    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed3a561_20ee_4f44_8c71_b92303967f57.slice/crio-34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f WatchSource:0}: Error finding container 34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f: Status 404 returned error can't find the container with id 34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f
Sep 08 18:51:23.276290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.276265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:23.276646 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.276627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:23.277364 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.277348    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:23.567214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.567114    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4" event={"ID":"7ed3a561-20ee-4f44-8c71-b92303967f57","Type":"ContainerStarted","Data":"34ddc7114d1553c9b5ddc349527ba71d2b917cbc8c4ca4161f95eb5c1e6c165d"}
Sep 08 18:51:23.567214 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.567171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4" event={"ID":"7ed3a561-20ee-4f44-8c71-b92303967f57","Type":"ContainerStarted","Data":"34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f"}
Sep 08 18:51:23.586594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:23.586546    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4" podStartSLOduration=1.586530485 podStartE2EDuration="1.586530485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:23.585705629 +0000 UTC m=+1658.037280774" watchObservedRunningTime="2026-09-08 18:51:23.586530485 +0000 UTC m=+1658.038105617"
Sep 08 18:51:24.574140 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:24.574103    2582 generic.go:356] "Generic (PLEG): container finished" podID="7ed3a561-20ee-4f44-8c71-b92303967f57" containerID="34ddc7114d1553c9b5ddc349527ba71d2b917cbc8c4ca4161f95eb5c1e6c165d" exitCode=0
Sep 08 18:51:24.574550 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:24.574145    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4" event={"ID":"7ed3a561-20ee-4f44-8c71-b92303967f57","Type":"ContainerDied","Data":"34ddc7114d1553c9b5ddc349527ba71d2b917cbc8c4ca4161f95eb5c1e6c165d"}
Sep 08 18:51:25.714794 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:25.714765    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:25.855036 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:25.854953    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca\") pod \"7ed3a561-20ee-4f44-8c71-b92303967f57\" (UID: \"7ed3a561-20ee-4f44-8c71-b92303967f57\") "
Sep 08 18:51:25.855306 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:25.855283    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ed3a561-20ee-4f44-8c71-b92303967f57" (UID: "7ed3a561-20ee-4f44-8c71-b92303967f57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:25.956662 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:25.956624    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ed3a561-20ee-4f44-8c71-b92303967f57-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:26.586557 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:26.586521    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"
Sep 08 18:51:26.586784 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:26.586522    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4" event={"ID":"7ed3a561-20ee-4f44-8c71-b92303967f57","Type":"ContainerDied","Data":"34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f"}
Sep 08 18:51:26.586784 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:26.586639    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f2e9d5de0650c1f32136e96b3b34cadf0668771ebbda8e0f36e5026e41838f"
Sep 08 18:51:28.161953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.161917    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:51:28.162430 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.162158    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" podUID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" containerName="payload-pre-processing" containerID="cri-o://ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad" gracePeriod=30
Sep 08 18:51:28.311098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.311063    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"]
Sep 08 18:51:28.316269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.316243    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-61f4b7-n7qt4"]
Sep 08 18:51:28.431550 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.431524    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:51:28.585267 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.585230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume\") pod \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") "
Sep 08 18:51:28.585450 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.585347    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt5m8\" (UniqueName: \"kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8\") pod \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\" (UID: \"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c\") "
Sep 08 18:51:28.585700 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.585668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" (UID: "96b8fd34-2bd9-4d56-8ae9-902d0d7c596c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:28.587597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.587572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8" (OuterVolumeSpecName: "kube-api-access-kt5m8") pod "96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" (UID: "96b8fd34-2bd9-4d56-8ae9-902d0d7c596c"). InnerVolumeSpecName "kube-api-access-kt5m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:51:28.601594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.601567    2582 generic.go:356] "Generic (PLEG): container finished" podID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" containerID="ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad" exitCode=0
Sep 08 18:51:28.601711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.601607    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"
Sep 08 18:51:28.601711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.601652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" event={"ID":"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c","Type":"ContainerDied","Data":"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"}
Sep 08 18:51:28.601711 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.601701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t" event={"ID":"96b8fd34-2bd9-4d56-8ae9-902d0d7c596c","Type":"ContainerDied","Data":"13fff722393ef0d8db67ef47de6acc32d174e2ae578b0c42f83023f47a0b8f63"}
Sep 08 18:51:28.601916 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.601750    2582 scope.go:117] "RemoveContainer" containerID="ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"
Sep 08 18:51:28.611261 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.611245    2582 scope.go:117] "RemoveContainer" containerID="ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"
Sep 08 18:51:28.611535 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:51:28.611518    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad\": container with ID starting with ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad not found: ID does not exist" containerID="ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"
Sep 08 18:51:28.611585 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.611544    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad"} err="failed to get container status \"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad\": rpc error: code = NotFound desc = could not find container \"ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad\": container with ID starting with ea1cd53a629d5ad8efbea0233cff542b0e24d54b11878f79041ae512b92d2cad not found: ID does not exist"
Sep 08 18:51:28.634223 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.634193    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:51:28.649054 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.649033    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-61f4b7-54c4bfb4d6-xnp8t"]
Sep 08 18:51:28.686610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.686534    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt5m8\" (UniqueName: \"kubernetes.io/projected/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-kube-api-access-kt5m8\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:28.686610 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:28.686562    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:30.099981 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:30.099942    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed3a561-20ee-4f44-8c71-b92303967f57" path="/var/lib/kubelet/pods/7ed3a561-20ee-4f44-8c71-b92303967f57/volumes"
Sep 08 18:51:30.100463 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:30.100283    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" path="/var/lib/kubelet/pods/96b8fd34-2bd9-4d56-8ae9-902d0d7c596c/volumes"
Sep 08 18:51:35.339107 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:35.339069    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:51:35.339503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:35.339339    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" containerID="cri-o://daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68" gracePeriod=30
Sep 08 18:51:38.279588 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:38.279550    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 18:51:38.279978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:38.279609    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:51:39.188060 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188023    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"]
Sep 08 18:51:39.188427 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188414    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed3a561-20ee-4f44-8c71-b92303967f57" containerName="revoke-keys"
Sep 08 18:51:39.188472 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188428    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed3a561-20ee-4f44-8c71-b92303967f57" containerName="revoke-keys"
Sep 08 18:51:39.188472 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188455    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" containerName="payload-pre-processing"
Sep 08 18:51:39.188472 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188461    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" containerName="payload-pre-processing"
Sep 08 18:51:39.188565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188513    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed3a561-20ee-4f44-8c71-b92303967f57" containerName="revoke-keys"
Sep 08 18:51:39.188565 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.188524    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b8fd34-2bd9-4d56-8ae9-902d0d7c596c" containerName="payload-pre-processing"
Sep 08 18:51:39.191295 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.191277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:39.194150 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.194124    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:51:39.198962 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.198934    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"]
Sep 08 18:51:39.283776 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.283706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl\" (UID: \"5e6a836a-4ac7-4beb-b393-20a617dfa684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:39.384681 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.384642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl\" (UID: \"5e6a836a-4ac7-4beb-b393-20a617dfa684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:39.385317 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.385293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl\" (UID: \"5e6a836a-4ac7-4beb-b393-20a617dfa684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:39.503122 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.503028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:39.779539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:39.779503    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"]
Sep 08 18:51:39.781702 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:51:39.781668    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6a836a_4ac7_4beb_b393_20a617dfa684.slice/crio-d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a WatchSource:0}: Error finding container d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a: Status 404 returned error can't find the container with id d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a
Sep 08 18:51:40.593923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.593895    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:40.666329 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.666292    2582 generic.go:356] "Generic (PLEG): container finished" podID="b3243def-385c-4827-9d1e-e9b48f928b70" containerID="daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68" exitCode=0
Sep 08 18:51:40.666528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.666331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerDied","Data":"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"}
Sep 08 18:51:40.666528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.666360    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"
Sep 08 18:51:40.666528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.666408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l" event={"ID":"b3243def-385c-4827-9d1e-e9b48f928b70","Type":"ContainerDied","Data":"d89f6c976217648ef284993abe482d41d87112074ac3307a85ae6f9d9a73c2c8"}
Sep 08 18:51:40.666528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.666442    2582 scope.go:117] "RemoveContainer" containerID="daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"
Sep 08 18:51:40.667993 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.667974    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e6a836a-4ac7-4beb-b393-20a617dfa684" containerID="3491f5c449a409c539d9419d599014a97c98c849afaad5c14546439b601a2165" exitCode=0
Sep 08 18:51:40.668109 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.668029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl" event={"ID":"5e6a836a-4ac7-4beb-b393-20a617dfa684","Type":"ContainerDied","Data":"3491f5c449a409c539d9419d599014a97c98c849afaad5c14546439b601a2165"}
Sep 08 18:51:40.668109 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.668079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl" event={"ID":"5e6a836a-4ac7-4beb-b393-20a617dfa684","Type":"ContainerStarted","Data":"d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a"}
Sep 08 18:51:40.675775 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.675759    2582 scope.go:117] "RemoveContainer" containerID="56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"
Sep 08 18:51:40.686895 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.686875    2582 scope.go:117] "RemoveContainer" containerID="daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"
Sep 08 18:51:40.687254 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:51:40.687231    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68\": container with ID starting with daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68 not found: ID does not exist" containerID="daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"
Sep 08 18:51:40.687332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.687263    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68"} err="failed to get container status \"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68\": rpc error: code = NotFound desc = could not find container \"daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68\": container with ID starting with daea5bd0379ccd51f96908f80f74bff8c7b6232645f505b145b0a49fd9e46d68 not found: ID does not exist"
Sep 08 18:51:40.687332 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.687283    2582 scope.go:117] "RemoveContainer" containerID="56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"
Sep 08 18:51:40.687587 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:51:40.687572    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda\": container with ID starting with 56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda not found: ID does not exist" containerID="56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"
Sep 08 18:51:40.687628 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.687591    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda"} err="failed to get container status \"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda\": rpc error: code = NotFound desc = could not find container \"56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda\": container with ID starting with 56892265b045a027c7f407c03d4f7ab0c1cdb885473cac6b71afb8bdd4661dda not found: ID does not exist"
Sep 08 18:51:40.697952 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.697929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.697966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.697993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-452zk\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698025 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698187    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698221    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:51:40.698276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698234    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:51:40.698276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698257    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698483 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698359    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:51:40.698483 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698382    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo\") pod \"b3243def-385c-4827-9d1e-e9b48f928b70\" (UID: \"b3243def-385c-4827-9d1e-e9b48f928b70\") "
Sep 08 18:51:40.698483 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698377    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:51:40.698626 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698475    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:40.698774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698747    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.699034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698778    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.699034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698792    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3243def-385c-4827-9d1e-e9b48f928b70-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.699034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698804    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.699034 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.698818    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.700290 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.700269    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk" (OuterVolumeSpecName: "kube-api-access-452zk") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "kube-api-access-452zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:51:40.700425 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.700404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:51:40.700499 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.700486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:51:40.700536 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.700502    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3243def-385c-4827-9d1e-e9b48f928b70" (UID: "b3243def-385c-4827-9d1e-e9b48f928b70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:51:40.799751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.799698    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3243def-385c-4827-9d1e-e9b48f928b70-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.799751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.799752    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-452zk\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-kube-api-access-452zk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.799948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.799765    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3243def-385c-4827-9d1e-e9b48f928b70-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.799948 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.799778    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3243def-385c-4827-9d1e-e9b48f928b70-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:40.991055 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.991024    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:51:40.995682 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:40.995658    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-61f4b7-openshift-default-db94c6f96-cdw2l"]
Sep 08 18:51:41.807197 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:41.807173    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:41.909837 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:41.909803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca\") pod \"5e6a836a-4ac7-4beb-b393-20a617dfa684\" (UID: \"5e6a836a-4ac7-4beb-b393-20a617dfa684\") "
Sep 08 18:51:41.910198 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:41.910176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e6a836a-4ac7-4beb-b393-20a617dfa684" (UID: "5e6a836a-4ac7-4beb-b393-20a617dfa684"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:51:42.010760 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:42.010639    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e6a836a-4ac7-4beb-b393-20a617dfa684-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:51:42.099209 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:42.099163    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" path="/var/lib/kubelet/pods/b3243def-385c-4827-9d1e-e9b48f928b70/volumes"
Sep 08 18:51:42.680231 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:42.680201    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"
Sep 08 18:51:42.680404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:42.680244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl" event={"ID":"5e6a836a-4ac7-4beb-b393-20a617dfa684","Type":"ContainerDied","Data":"d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a"}
Sep 08 18:51:42.680404 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:42.680292    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05099f406dbc47ea5aea5b67ed9b73b9479ea530b8187cefb47a0762815fe9a"
Sep 08 18:51:46.404308 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:46.404204    2582 scope.go:117] "RemoveContainer" containerID="97435763a3ad7f19e0791a024438758ab68112230eeeed19cbc1fe47b99054d5"
Sep 08 18:51:55.444893 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:55.444861    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"]
Sep 08 18:51:55.449418 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:55.449390    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bc23ef-22nsl"]
Sep 08 18:51:56.099843 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:51:56.099808    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e6a836a-4ac7-4beb-b393-20a617dfa684" path="/var/lib/kubelet/pods/5e6a836a-4ac7-4beb-b393-20a617dfa684/volumes"
Sep 08 18:52:00.790587 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:00.790544    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:52:15.810589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:15.810521    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" podUID="56e4360d-011d-491c-bb67-add0765013b4" containerName="main" containerID="cri-o://15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7" gracePeriod=45
Sep 08 18:52:16.068986 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.068925    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:52:16.127368 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127331    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127368 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127453    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127480    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6l8t\" (UniqueName: \"kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127514    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127623 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache\") pod \"56e4360d-011d-491c-bb67-add0765013b4\" (UID: \"56e4360d-011d-491c-bb67-add0765013b4\") "
Sep 08 18:52:16.127961 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127935    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home" (OuterVolumeSpecName: "home") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:52:16.128047 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.127946    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:52:16.128279 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.128182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:52:16.129940 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.129918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm" (OuterVolumeSpecName: "dshm") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:52:16.130388 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.130359    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:52:16.130486 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.130425    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t" (OuterVolumeSpecName: "kube-api-access-j6l8t") pod "56e4360d-011d-491c-bb67-add0765013b4" (UID: "56e4360d-011d-491c-bb67-add0765013b4"). InnerVolumeSpecName "kube-api-access-j6l8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:52:16.228814 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228785    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.228814 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228810    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.228814 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228818    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.229050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228826    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e4360d-011d-491c-bb67-add0765013b4-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.229050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228835    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6l8t\" (UniqueName: \"kubernetes.io/projected/56e4360d-011d-491c-bb67-add0765013b4-kube-api-access-j6l8t\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.229050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.228843    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e4360d-011d-491c-bb67-add0765013b4-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:52:16.859516 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.859480    2582 generic.go:356] "Generic (PLEG): container finished" podID="56e4360d-011d-491c-bb67-add0765013b4" containerID="15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7" exitCode=0
Sep 08 18:52:16.860069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.859525    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"
Sep 08 18:52:16.860069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.859573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" event={"ID":"56e4360d-011d-491c-bb67-add0765013b4","Type":"ContainerDied","Data":"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"}
Sep 08 18:52:16.860069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.859635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b" event={"ID":"56e4360d-011d-491c-bb67-add0765013b4","Type":"ContainerDied","Data":"71a27113024eabde8b50c29cdfaf4c69a001168717f6d983d0d7302caa22f795"}
Sep 08 18:52:16.860069 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.859661    2582 scope.go:117] "RemoveContainer" containerID="15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"
Sep 08 18:52:16.869239 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.869223    2582 scope.go:117] "RemoveContainer" containerID="15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"
Sep 08 18:52:16.869518 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:52:16.869498    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7\": container with ID starting with 15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7 not found: ID does not exist" containerID="15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"
Sep 08 18:52:16.869608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.869526    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7"} err="failed to get container status \"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7\": rpc error: code = NotFound desc = could not find container \"15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7\": container with ID starting with 15880c9e8323e5d603f3c37fddc44aab8155f6b1b243e1fc611956f2b5aeabc7 not found: ID does not exist"
Sep 08 18:52:16.884459 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.884431    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:52:16.886249 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:16.886229    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bc23ef/ipp-route-bc23ef-kserve-64945657c5-44f9b"]
Sep 08 18:52:18.105020 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:18.104983    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e4360d-011d-491c-bb67-add0765013b4" path="/var/lib/kubelet/pods/56e4360d-011d-491c-bb67-add0765013b4/volumes"
Sep 08 18:52:58.720859 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.720828    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"]
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721194    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e6a836a-4ac7-4beb-b393-20a617dfa684" containerName="revoke-keys"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721205    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e6a836a-4ac7-4beb-b393-20a617dfa684" containerName="revoke-keys"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721218    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e4360d-011d-491c-bb67-add0765013b4" containerName="main"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721223    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e4360d-011d-491c-bb67-add0765013b4" containerName="main"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721230    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721236    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721250    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.721277 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721256    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.721560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721317    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.721560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721325    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e6a836a-4ac7-4beb-b393-20a617dfa684" containerName="revoke-keys"
Sep 08 18:52:58.721560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721332    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e4360d-011d-491c-bb67-add0765013b4" containerName="main"
Sep 08 18:52:58.721560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.721339    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3243def-385c-4827-9d1e-e9b48f928b70" containerName="istio-proxy"
Sep 08 18:52:58.724133 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.724118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:52:58.726778 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.726758    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:52:58.732466 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.732433    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"]
Sep 08 18:52:58.917017 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:58.916981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk\" (UID: \"8c127c60-c289-40d7-aa98-20d92cbe15b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:52:59.017648 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:59.017562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk\" (UID: \"8c127c60-c289-40d7-aa98-20d92cbe15b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:52:59.018196 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:59.018176    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk\" (UID: \"8c127c60-c289-40d7-aa98-20d92cbe15b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:52:59.035564 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:59.035543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:52:59.306295 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:52:59.306267    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"]
Sep 08 18:52:59.307746 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:52:59.307693    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c127c60_c289_40d7_aa98_20d92cbe15b4.slice/crio-11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a WatchSource:0}: Error finding container 11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a: Status 404 returned error can't find the container with id 11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a
Sep 08 18:53:00.081213 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:00.081176    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c127c60-c289-40d7-aa98-20d92cbe15b4" containerID="b8fe2274773ef72deef2676af893af834716bfc99ccb572ead58022c44d74268" exitCode=0
Sep 08 18:53:00.081620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:00.081262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk" event={"ID":"8c127c60-c289-40d7-aa98-20d92cbe15b4","Type":"ContainerDied","Data":"b8fe2274773ef72deef2676af893af834716bfc99ccb572ead58022c44d74268"}
Sep 08 18:53:00.081620 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:00.081311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk" event={"ID":"8c127c60-c289-40d7-aa98-20d92cbe15b4","Type":"ContainerStarted","Data":"11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a"}
Sep 08 18:53:01.223339 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:01.223312    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:53:01.232631 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:01.232599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca\") pod \"8c127c60-c289-40d7-aa98-20d92cbe15b4\" (UID: \"8c127c60-c289-40d7-aa98-20d92cbe15b4\") "
Sep 08 18:53:01.232956 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:01.232934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c127c60-c289-40d7-aa98-20d92cbe15b4" (UID: "8c127c60-c289-40d7-aa98-20d92cbe15b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:01.333540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:01.333496    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c127c60-c289-40d7-aa98-20d92cbe15b4-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:02.093652 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:02.093622    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"
Sep 08 18:53:02.106263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:02.106232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk" event={"ID":"8c127c60-c289-40d7-aa98-20d92cbe15b4","Type":"ContainerDied","Data":"11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a"}
Sep 08 18:53:02.106263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:02.106264    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c1cce35c312380bebf208b149aa7e432703f2326ff892fea22cb57fd4d542a"
Sep 08 18:53:09.386599 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.386564    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:53:09.387047 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.386790    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" podUID="78359ff6-262c-4b8b-8034-79aa77618678" containerName="payload-pre-processing" containerID="cri-o://3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd" gracePeriod=30
Sep 08 18:53:09.539141 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.539108    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"]
Sep 08 18:53:09.542849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.542820    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-302afb-dp4bk"]
Sep 08 18:53:09.665877 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.665850    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:53:09.701935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.701902    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv2k5\" (UniqueName: \"kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5\") pod \"78359ff6-262c-4b8b-8034-79aa77618678\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") "
Sep 08 18:53:09.702110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.701947    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume\") pod \"78359ff6-262c-4b8b-8034-79aa77618678\" (UID: \"78359ff6-262c-4b8b-8034-79aa77618678\") "
Sep 08 18:53:09.702359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.702332    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78359ff6-262c-4b8b-8034-79aa77618678" (UID: "78359ff6-262c-4b8b-8034-79aa77618678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:09.704284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.704261    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5" (OuterVolumeSpecName: "kube-api-access-pv2k5") pod "78359ff6-262c-4b8b-8034-79aa77618678" (UID: "78359ff6-262c-4b8b-8034-79aa77618678"). InnerVolumeSpecName "kube-api-access-pv2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:09.802954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.802916    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2k5\" (UniqueName: \"kubernetes.io/projected/78359ff6-262c-4b8b-8034-79aa77618678-kube-api-access-pv2k5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:09.802954 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:09.802949    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78359ff6-262c-4b8b-8034-79aa77618678-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:10.099364 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.099284    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c127c60-c289-40d7-aa98-20d92cbe15b4" path="/var/lib/kubelet/pods/8c127c60-c289-40d7-aa98-20d92cbe15b4/volumes"
Sep 08 18:53:10.133497 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.133460    2582 generic.go:356] "Generic (PLEG): container finished" podID="78359ff6-262c-4b8b-8034-79aa77618678" containerID="3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd" exitCode=0
Sep 08 18:53:10.133668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.133499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" event={"ID":"78359ff6-262c-4b8b-8034-79aa77618678","Type":"ContainerDied","Data":"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"}
Sep 08 18:53:10.133668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.133524    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"
Sep 08 18:53:10.133668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.133570    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg" event={"ID":"78359ff6-262c-4b8b-8034-79aa77618678","Type":"ContainerDied","Data":"014bd2094316c07b5ab8acff6626834dd0145dd8f14ff6833d0e710f7b1a7d22"}
Sep 08 18:53:10.133668 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.133601    2582 scope.go:117] "RemoveContainer" containerID="3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"
Sep 08 18:53:10.142257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.142238    2582 scope.go:117] "RemoveContainer" containerID="3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"
Sep 08 18:53:10.142554 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:10.142537    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd\": container with ID starting with 3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd not found: ID does not exist" containerID="3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"
Sep 08 18:53:10.142596 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.142562    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd"} err="failed to get container status \"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd\": rpc error: code = NotFound desc = could not find container \"3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd\": container with ID starting with 3fcad936b3992595e68cb4e11939a3b04a3f54d2cb8d27609daf27120cdf14fd not found: ID does not exist"
Sep 08 18:53:10.152177 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.152147    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:53:10.154087 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:10.154068    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-302afb-7bf5f4f966-wpkmg"]
Sep 08 18:53:12.104953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:12.104908    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78359ff6-262c-4b8b-8034-79aa77618678" path="/var/lib/kubelet/pods/78359ff6-262c-4b8b-8034-79aa77618678/volumes"
Sep 08 18:53:15.959110 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:15.959076    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:53:15.959574 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:15.959289    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy" containerID="cri-o://26b306a7f0605eec48f823c36aa8851ecd86131aefc64893830760839993bf20" gracePeriod=30
Sep 08 18:53:21.193624 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.193592    2582 generic.go:356] "Generic (PLEG): container finished" podID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerID="26b306a7f0605eec48f823c36aa8851ecd86131aefc64893830760839993bf20" exitCode=0
Sep 08 18:53:21.194033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.193655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerDied","Data":"26b306a7f0605eec48f823c36aa8851ecd86131aefc64893830760839993bf20"}
Sep 08 18:53:21.194033 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.193771    2582 scope.go:117] "RemoveContainer" containerID="ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760"
Sep 08 18:53:21.215100 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:21.215078    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760\": container with ID starting with ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760 not found: ID does not exist" containerID="ec52f911228d900296b8fd3f11ea630bbcb816d1c7a0defa7d04cb02eb379760"
Sep 08 18:53:21.218276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.218256    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:53:21.303342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303265    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303342 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303322    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgq9t\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303353    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303393    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303427    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303464    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303518 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303566    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token\") pod \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\" (UID: \"6f55d1c6-ab0b-4103-b7a1-b0899c9820da\") "
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303568    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303650    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303691    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:21.303779 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303717    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:21.304176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303931    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.304176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303943    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:21.304176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303950    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.304176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303978    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.304176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.303988    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.305780 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.305759    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:21.305891 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.305859    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:21.305937 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.305895    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t" (OuterVolumeSpecName: "kube-api-access-cgq9t") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "kube-api-access-cgq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:21.305937 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.305896    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f55d1c6-ab0b-4103-b7a1-b0899c9820da" (UID: "6f55d1c6-ab0b-4103-b7a1-b0899c9820da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:53:21.404540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.404508    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.404540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.404534    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.404540 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.404543    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgq9t\" (UniqueName: \"kubernetes.io/projected/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-kube-api-access-cgq9t\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.404793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.404551    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:21.404793 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:21.404560    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f55d1c6-ab0b-4103-b7a1-b0899c9820da-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:22.199843 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:22.199813    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"
Sep 08 18:53:22.200262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:22.199813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp" event={"ID":"6f55d1c6-ab0b-4103-b7a1-b0899c9820da","Type":"ContainerDied","Data":"c68c9dc50fec6473baf5706c2ce0e5a3b0d0544ab5b1a746aa06c60615655dfb"}
Sep 08 18:53:22.200262 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:22.199942    2582 scope.go:117] "RemoveContainer" containerID="26b306a7f0605eec48f823c36aa8851ecd86131aefc64893830760839993bf20"
Sep 08 18:53:22.220281 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:22.220259    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:53:22.224984 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:22.224949    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-302afb-openshift-default-58c44d6bbd-2clrp"]
Sep 08 18:53:24.099002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:24.098968    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" path="/var/lib/kubelet/pods/6f55d1c6-ab0b-4103-b7a1-b0899c9820da/volumes"
Sep 08 18:53:26.772139 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:26.772103    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:53:26.772577 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:26.772339    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" podUID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" containerName="payload-processing" containerID="cri-o://1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d" gracePeriod=30
Sep 08 18:53:26.777481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:26.777451    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:53:26.777868 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:26.777830    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" podUID="3fc50723-0425-4388-9781-9b156156da5c" containerName="payload-pre-processing" containerID="cri-o://556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896" gracePeriod=30
Sep 08 18:53:27.057149 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.057120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:53:27.060999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.060977    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:53:27.157550 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.157516    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume\") pod \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") "
Sep 08 18:53:27.157781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.157577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dpwc\" (UniqueName: \"kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc\") pod \"3fc50723-0425-4388-9781-9b156156da5c\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") "
Sep 08 18:53:27.157781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.157611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmrn\" (UniqueName: \"kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn\") pod \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\" (UID: \"eb36e503-d790-4cca-916f-12fd0d5d6e8c\") "
Sep 08 18:53:27.157781 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.157683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume\") pod \"3fc50723-0425-4388-9781-9b156156da5c\" (UID: \"3fc50723-0425-4388-9781-9b156156da5c\") "
Sep 08 18:53:27.158067 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.158036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb36e503-d790-4cca-916f-12fd0d5d6e8c" (UID: "eb36e503-d790-4cca-916f-12fd0d5d6e8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:27.158180 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.158084    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fc50723-0425-4388-9781-9b156156da5c" (UID: "3fc50723-0425-4388-9781-9b156156da5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:27.160073 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.160040    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc" (OuterVolumeSpecName: "kube-api-access-8dpwc") pod "3fc50723-0425-4388-9781-9b156156da5c" (UID: "3fc50723-0425-4388-9781-9b156156da5c"). InnerVolumeSpecName "kube-api-access-8dpwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:27.160205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.160180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn" (OuterVolumeSpecName: "kube-api-access-jfmrn") pod "eb36e503-d790-4cca-916f-12fd0d5d6e8c" (UID: "eb36e503-d790-4cca-916f-12fd0d5d6e8c"). InnerVolumeSpecName "kube-api-access-jfmrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:27.228113 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.228078    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" containerID="1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d" exitCode=0
Sep 08 18:53:27.228302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.228123    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"
Sep 08 18:53:27.228302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.228166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" event={"ID":"eb36e503-d790-4cca-916f-12fd0d5d6e8c","Type":"ContainerDied","Data":"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"}
Sep 08 18:53:27.228302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.228220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757" event={"ID":"eb36e503-d790-4cca-916f-12fd0d5d6e8c","Type":"ContainerDied","Data":"fcc53867d7e854eac55981791f657514c9febec1b95c8704a752996bd07bf370"}
Sep 08 18:53:27.228302 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.228247    2582 scope.go:117] "RemoveContainer" containerID="1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"
Sep 08 18:53:27.229586 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.229565    2582 generic.go:356] "Generic (PLEG): container finished" podID="3fc50723-0425-4388-9781-9b156156da5c" containerID="556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896" exitCode=0
Sep 08 18:53:27.229691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.229613    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"
Sep 08 18:53:27.229691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.229647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" event={"ID":"3fc50723-0425-4388-9781-9b156156da5c","Type":"ContainerDied","Data":"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"}
Sep 08 18:53:27.229691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.229688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb" event={"ID":"3fc50723-0425-4388-9781-9b156156da5c","Type":"ContainerDied","Data":"2abb9381d4f59d2fe5866b4b8af2718e6df0e72a1bb099c266f419d42c865782"}
Sep 08 18:53:27.237063 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.237038    2582 scope.go:117] "RemoveContainer" containerID="1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"
Sep 08 18:53:27.237338 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:27.237316    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d\": container with ID starting with 1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d not found: ID does not exist" containerID="1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"
Sep 08 18:53:27.237413 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.237345    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d"} err="failed to get container status \"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d\": rpc error: code = NotFound desc = could not find container \"1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d\": container with ID starting with 1dcceae533ab467a502d595beea46eb003f716526e655c6ddb888be6c34efc9d not found: ID does not exist"
Sep 08 18:53:27.237413 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.237364    2582 scope.go:117] "RemoveContainer" containerID="556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"
Sep 08 18:53:27.245393 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.245376    2582 scope.go:117] "RemoveContainer" containerID="556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"
Sep 08 18:53:27.245682 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:27.245662    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896\": container with ID starting with 556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896 not found: ID does not exist" containerID="556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"
Sep 08 18:53:27.245798 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.245687    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896"} err="failed to get container status \"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896\": rpc error: code = NotFound desc = could not find container \"556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896\": container with ID starting with 556532f1e6c85132b69dfdd7e53ee210fb331821224bb616a62458f6f34d1896 not found: ID does not exist"
Sep 08 18:53:27.258800 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.258775    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:53:27.259100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.259073    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpwc\" (UniqueName: \"kubernetes.io/projected/3fc50723-0425-4388-9781-9b156156da5c-kube-api-access-8dpwc\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.259100 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.259097    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmrn\" (UniqueName: \"kubernetes.io/projected/eb36e503-d790-4cca-916f-12fd0d5d6e8c-kube-api-access-jfmrn\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.259259 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.259107    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc50723-0425-4388-9781-9b156156da5c-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.259259 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.259116    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb36e503-d790-4cca-916f-12fd0d5d6e8c-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:27.262010 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.261987    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-792c35-58c4747d8f-cg757"]
Sep 08 18:53:27.273070 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.273042    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:53:27.275153 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:27.275134    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-792c35-7857564994-tk7nb"]
Sep 08 18:53:28.099659 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:28.099629    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc50723-0425-4388-9781-9b156156da5c" path="/var/lib/kubelet/pods/3fc50723-0425-4388-9781-9b156156da5c/volumes"
Sep 08 18:53:28.100065 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:28.100013    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" path="/var/lib/kubelet/pods/eb36e503-d790-4cca-916f-12fd0d5d6e8c/volumes"
Sep 08 18:53:33.745953 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:33.745910    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:53:33.746547 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:33.746180    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" containerID="cri-o://7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0" gracePeriod=30
Sep 08 18:53:34.136823 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.136792    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"]
Sep 08 18:53:34.137176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137157    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78359ff6-262c-4b8b-8034-79aa77618678" containerName="payload-pre-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137178    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78359ff6-262c-4b8b-8034-79aa77618678" containerName="payload-pre-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137197    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" containerName="payload-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137203    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" containerName="payload-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137211    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc50723-0425-4388-9781-9b156156da5c" containerName="payload-pre-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137216    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc50723-0425-4388-9781-9b156156da5c" containerName="payload-pre-processing"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137224    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137231    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137241    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:53:34.137252 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137250    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137260    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c127c60-c289-40d7-aa98-20d92cbe15b4" containerName="revoke-keys"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137266    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c127c60-c289-40d7-aa98-20d92cbe15b4" containerName="revoke-keys"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137324    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78359ff6-262c-4b8b-8034-79aa77618678" containerName="payload-pre-processing"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137334    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c127c60-c289-40d7-aa98-20d92cbe15b4" containerName="revoke-keys"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137340    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc50723-0425-4388-9781-9b156156da5c" containerName="payload-pre-processing"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137347    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:53:34.137713 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.137354    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb36e503-d790-4cca-916f-12fd0d5d6e8c" containerName="payload-processing"
Sep 08 18:53:34.141858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.141838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:34.145024 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.144956    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 18:53:34.147307 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.147280    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"]
Sep 08 18:53:34.218742 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.218684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp\" (UID: \"647fab81-f089-48db-9e36-78ef582a13e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:34.319905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.319865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp\" (UID: \"647fab81-f089-48db-9e36-78ef582a13e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:34.320560 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.320538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp\" (UID: \"647fab81-f089-48db-9e36-78ef582a13e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:34.454935 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.454842    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:34.724911 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.724885    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"]
Sep 08 18:53:34.727156 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:53:34.727128    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647fab81_f089_48db_9e36_78ef582a13e7.slice/crio-5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a WatchSource:0}: Error finding container 5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a: Status 404 returned error can't find the container with id 5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a
Sep 08 18:53:34.885316 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.885281    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 18:53:34.885694 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:34.885376    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 18:53:35.279420 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:35.279372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp" event={"ID":"647fab81-f089-48db-9e36-78ef582a13e7","Type":"ContainerStarted","Data":"6ed2ee858251f4115d723385f91beecaee92d65558b56990ae8f0e8186556374"}
Sep 08 18:53:35.279608 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:35.279469    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp" event={"ID":"647fab81-f089-48db-9e36-78ef582a13e7","Type":"ContainerStarted","Data":"5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a"}
Sep 08 18:53:35.296618 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:35.296551    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp" podStartSLOduration=1.2965357530000001 podStartE2EDuration="1.296535753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:35.293703223 +0000 UTC m=+1789.745278357" watchObservedRunningTime="2026-09-08 18:53:35.296535753 +0000 UTC m=+1789.748110932"
Sep 08 18:53:36.289507 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:36.289471    2582 generic.go:356] "Generic (PLEG): container finished" podID="647fab81-f089-48db-9e36-78ef582a13e7" containerID="6ed2ee858251f4115d723385f91beecaee92d65558b56990ae8f0e8186556374" exitCode=0
Sep 08 18:53:36.289928 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:36.289535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp" event={"ID":"647fab81-f089-48db-9e36-78ef582a13e7","Type":"ContainerDied","Data":"6ed2ee858251f4115d723385f91beecaee92d65558b56990ae8f0e8186556374"}
Sep 08 18:53:37.426594 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:37.426569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:37.550284 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:37.550193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca\") pod \"647fab81-f089-48db-9e36-78ef582a13e7\" (UID: \"647fab81-f089-48db-9e36-78ef582a13e7\") "
Sep 08 18:53:37.550554 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:37.550530    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "647fab81-f089-48db-9e36-78ef582a13e7" (UID: "647fab81-f089-48db-9e36-78ef582a13e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:37.651557 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:37.651521    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647fab81-f089-48db-9e36-78ef582a13e7-maas-api-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:38.301227 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:38.301198    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"
Sep 08 18:53:38.301394 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:38.301248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp" event={"ID":"647fab81-f089-48db-9e36-78ef582a13e7","Type":"ContainerDied","Data":"5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a"}
Sep 08 18:53:38.301394 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:38.301293    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5493a8c073e545d4e0ac2fc0161bfdb32cfdfc7d5995e76d777c59506714266a"
Sep 08 18:53:38.998172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:38.998145    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:53:39.064522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6snx\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064493    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064522 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064517    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064542    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064624    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064687    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.064902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064772    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket\") pod \"47b5272a-bae3-446a-a66a-b5e80fef86d8\" (UID: \"47b5272a-bae3-446a-a66a-b5e80fef86d8\") "
Sep 08 18:53:39.065164 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.064981    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:39.065164 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:39.065276 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:39.065334 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065264    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.065462 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065402    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b5272a-bae3-446a-a66a-b5e80fef86d8-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.065639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065331    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:39.065774 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.065564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:39.067088 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.067050    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 08 18:53:39.067187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.067111    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx" (OuterVolumeSpecName: "kube-api-access-n6snx") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "kube-api-access-n6snx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:39.067187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.067133    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:39.067677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.067657    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47b5272a-bae3-446a-a66a-b5e80fef86d8" (UID: "47b5272a-bae3-446a-a66a-b5e80fef86d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:53:39.166898 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166862    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6snx\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-kube-api-access-n6snx\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.166898 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166891    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.166898 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166901    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.166898 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166908    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.167170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166916    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.167170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166924    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.167170 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.166932    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b5272a-bae3-446a-a66a-b5e80fef86d8-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:39.309293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.309256    2582 generic.go:356] "Generic (PLEG): container finished" podID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerID="7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0" exitCode=0
Sep 08 18:53:39.309439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.309309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"
Sep 08 18:53:39.309439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.309342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerDied","Data":"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"}
Sep 08 18:53:39.309439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.309396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm" event={"ID":"47b5272a-bae3-446a-a66a-b5e80fef86d8","Type":"ContainerDied","Data":"c44a06eb0b6904f1b2ee7a99e457f220e3989223d3f7fbe63179927c8b58e751"}
Sep 08 18:53:39.309439 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.309421    2582 scope.go:117] "RemoveContainer" containerID="7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"
Sep 08 18:53:39.318992 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.318969    2582 scope.go:117] "RemoveContainer" containerID="b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"
Sep 08 18:53:39.327330 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.327314    2582 scope.go:117] "RemoveContainer" containerID="7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"
Sep 08 18:53:39.327582 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:39.327565    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0\": container with ID starting with 7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0 not found: ID does not exist" containerID="7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"
Sep 08 18:53:39.327621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.327592    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0"} err="failed to get container status \"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0\": rpc error: code = NotFound desc = could not find container \"7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0\": container with ID starting with 7234259542c14c2b1373fd0406b37c3731fb1c9d0aa9b1d362a885c2ca2b45d0 not found: ID does not exist"
Sep 08 18:53:39.327621 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.327606    2582 scope.go:117] "RemoveContainer" containerID="b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"
Sep 08 18:53:39.327859 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:39.327843    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97\": container with ID starting with b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97 not found: ID does not exist" containerID="b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"
Sep 08 18:53:39.327904 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.327864    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97"} err="failed to get container status \"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97\": rpc error: code = NotFound desc = could not find container \"b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97\": container with ID starting with b2e96913408e21c0ead2adab22622bb249f8d80508a36a0b48b7e46dd2cf5b97 not found: ID does not exist"
Sep 08 18:53:39.333707 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.333685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:53:39.337506 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:39.337489    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-792c35-openshift-default-54cc686cc4-s72pm"]
Sep 08 18:53:40.098852 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:40.098818    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" path="/var/lib/kubelet/pods/47b5272a-bae3-446a-a66a-b5e80fef86d8/volumes"
Sep 08 18:53:44.646433 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:44.646402    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:53:44.646841 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:44.646586    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" podUID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" containerName="payload-processing" containerID="cri-o://1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0" gracePeriod=30
Sep 08 18:53:44.806324 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:44.806270    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"]
Sep 08 18:53:44.808031 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:44.808002    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6255f2-85nfp"]
Sep 08 18:53:44.907263 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:44.907181    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:53:45.021923 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.021887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6bq\" (UniqueName: \"kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq\") pod \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") "
Sep 08 18:53:45.022179 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.022014    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume\") pod \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\" (UID: \"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a\") "
Sep 08 18:53:45.022455 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.022422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" (UID: "b2b216c7-f6f5-4f15-bd2f-9222bb5f167a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:53:45.024213 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.024191    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq" (OuterVolumeSpecName: "kube-api-access-sh6bq") pod "b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" (UID: "b2b216c7-f6f5-4f15-bd2f-9222bb5f167a"). InnerVolumeSpecName "kube-api-access-sh6bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:53:45.123176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.123136    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-plugins-config-volume\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.123176 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.123169    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6bq\" (UniqueName: \"kubernetes.io/projected/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a-kube-api-access-sh6bq\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:53:45.339907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.339822    2582 generic.go:356] "Generic (PLEG): container finished" podID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" containerID="1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0" exitCode=0
Sep 08 18:53:45.339907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.339862    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"
Sep 08 18:53:45.339907 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.339870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" event={"ID":"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a","Type":"ContainerDied","Data":"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"}
Sep 08 18:53:45.340148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.339917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s" event={"ID":"b2b216c7-f6f5-4f15-bd2f-9222bb5f167a","Type":"ContainerDied","Data":"fd9162a8c503e8d06f4ee52b1687940c11d23a923469801df2bc015e25cc70c8"}
Sep 08 18:53:45.340148 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.339941    2582 scope.go:117] "RemoveContainer" containerID="1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"
Sep 08 18:53:45.349398 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.349378    2582 scope.go:117] "RemoveContainer" containerID="1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"
Sep 08 18:53:45.349675 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:53:45.349653    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0\": container with ID starting with 1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0 not found: ID does not exist" containerID="1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"
Sep 08 18:53:45.349718 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.349684    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0"} err="failed to get container status \"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0\": rpc error: code = NotFound desc = could not find container \"1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0\": container with ID starting with 1d3f74e073cbee597d5265f6753ed5acd73513539d1c69b8bb40f188859a9fc0 not found: ID does not exist"
Sep 08 18:53:45.362667 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.362639    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:53:45.368077 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:45.368056    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6255f2-7bd6bfbb69-ht44s"]
Sep 08 18:53:46.105243 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:46.105211    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647fab81-f089-48db-9e36-78ef582a13e7" path="/var/lib/kubelet/pods/647fab81-f089-48db-9e36-78ef582a13e7/volumes"
Sep 08 18:53:46.105759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:46.105716    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" path="/var/lib/kubelet/pods/b2b216c7-f6f5-4f15-bd2f-9222bb5f167a/volumes"
Sep 08 18:53:46.448465 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:53:46.448406    2582 scope.go:117] "RemoveContainer" containerID="875293b9442ba5d27973bdc1247c7a5a82b9d9630ee3dc1562b8d8092c8b24fe"
Sep 08 18:54:46.493611 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:54:46.493582    2582 scope.go:117] "RemoveContainer" containerID="c46a9595445e395567e04521ef1a618a023dd3f02bc3bc6598bc8b0a23120da5"
Sep 08 18:54:46.503289 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:54:46.503238    2582 scope.go:117] "RemoveContainer" containerID="57e34dd3ffd2eb52cf7086d2570ebe4cd19603b7b36902c26b41cf134aaf34a5"
Sep 08 18:57:46.540905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:57:46.540865    2582 scope.go:117] "RemoveContainer" containerID="34ddc7114d1553c9b5ddc349527ba71d2b917cbc8c4ca4161f95eb5c1e6c165d"
Sep 08 18:57:46.550138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:57:46.550103    2582 scope.go:117] "RemoveContainer" containerID="3491f5c449a409c539d9419d599014a97c98c849afaad5c14546439b601a2165"
Sep 08 18:58:16.022303 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022267    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:58:16.022849 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022824    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" containerName="payload-processing"
Sep 08 18:58:16.023003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022945    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" containerName="payload-processing"
Sep 08 18:58:16.023003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:16.023003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022986    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:16.023003 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.022999    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647fab81-f089-48db-9e36-78ef582a13e7" containerName="revoke-keys"
Sep 08 18:58:16.023211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.023008    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="647fab81-f089-48db-9e36-78ef582a13e7" containerName="revoke-keys"
Sep 08 18:58:16.023211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.023139    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b216c7-f6f5-4f15-bd2f-9222bb5f167a" containerName="payload-processing"
Sep 08 18:58:16.023211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.023153    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:16.023211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.023164    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f55d1c6-ab0b-4103-b7a1-b0899c9820da" containerName="istio-proxy"
Sep 08 18:58:16.023211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.023172    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="647fab81-f089-48db-9e36-78ef582a13e7" containerName="revoke-keys"
Sep 08 18:58:16.026201 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.026180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.029172 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.029152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-98edebc4-kserve-self-signed-certs\""
Sep 08 18:58:16.041818 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.041787    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:58:16.077604 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.077751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.077751 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.077832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.077832 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.077905 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.077890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-865qc\" (UniqueName: \"kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178267 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178457 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178676 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-865qc\" (UniqueName: \"kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178778 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.178858 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.178832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.180737 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.180705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.180946 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.180927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.187682 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.187661    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-865qc\" (UniqueName: \"kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc\") pod \"e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") " pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.336383 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.336279    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:16.610269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.610236    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:58:16.612769 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:58:16.612745    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd47c6373_6997_4b41_a561_4123456eefc5.slice/crio-73327086d893eb49e352231400fd646b97a74d39079882c8fb9e7d3fb4d2d402 WatchSource:0}: Error finding container 73327086d893eb49e352231400fd646b97a74d39079882c8fb9e7d3fb4d2d402: Status 404 returned error can't find the container with id 73327086d893eb49e352231400fd646b97a74d39079882c8fb9e7d3fb4d2d402
Sep 08 18:58:16.636163 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:16.636139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" event={"ID":"d47c6373-6997-4b41-a561-4123456eefc5","Type":"ContainerStarted","Data":"73327086d893eb49e352231400fd646b97a74d39079882c8fb9e7d3fb4d2d402"}
Sep 08 18:58:17.642359 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:17.642318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" event={"ID":"d47c6373-6997-4b41-a561-4123456eefc5","Type":"ContainerStarted","Data":"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"}
Sep 08 18:58:26.336829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:26.336784    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:26.336829 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:26.336840    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:26.349481 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:26.349454    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:26.370239 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:26.370186    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" podStartSLOduration=10.37017055 podStartE2EDuration="10.37017055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:17.660504683 +0000 UTC m=+2072.112079816" watchObservedRunningTime="2026-09-08 18:58:26.37017055 +0000 UTC m=+2080.821745719"
Sep 08 18:58:26.696775 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:26.696746    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:58:32.241484 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.241447    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:58:32.241903 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.241809    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:32.241903 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.241821    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:32.241903 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.241890    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b5272a-bae3-446a-a66a-b5e80fef86d8" containerName="istio-proxy"
Sep 08 18:58:32.245151 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.245129    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.248117 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.248096    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-98edebc4-kserve-self-signed-certs\""
Sep 08 18:58:32.260915 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.260884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:58:32.310157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tjj\" (UniqueName: \"kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.310157 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.310360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.310360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310273    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.310360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.310360 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.310357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411384 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411639 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tjj\" (UniqueName: \"kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411902 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.411965 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.411908    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.412230 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.412208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.412378 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.412359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.413901 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.413870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.414310 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.414290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.424528 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.424499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tjj\" (UniqueName: \"kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj\") pod \"e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") " pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.555806 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.555695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:32.833421 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:32.830854    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:58:33.721531 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:33.721498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" event={"ID":"e2d417ed-9d09-4c7f-b639-bc5d68aab78d","Type":"ContainerStarted","Data":"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"}
Sep 08 18:58:33.722052 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:33.721540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" event={"ID":"e2d417ed-9d09-4c7f-b639-bc5d68aab78d","Type":"ContainerStarted","Data":"c1bccb92a6cb7e91cb3b356260404f34082dfe3051ac85f37924ddd7a208cd2d"}
Sep 08 18:58:33.741908 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:33.741855    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" podStartSLOduration=1.7418374220000001 podStartE2EDuration="1.741837422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:58:33.738610265 +0000 UTC m=+2088.190185394" watchObservedRunningTime="2026-09-08 18:58:33.741837422 +0000 UTC m=+2088.193412553"
Sep 08 18:58:42.556777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:42.556705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:42.556777 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:42.556792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:42.569441 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:42.569405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:58:42.775759 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:58:42.775712    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:59:09.673883 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:09.673849    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:59:10.998795 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:10.998759    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:59:24.694211 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.694166    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" podUID="d47c6373-6997-4b41-a561-4123456eefc5" containerName="main" containerID="cri-o://231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15" gracePeriod=45
Sep 08 18:59:24.948205 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.948137    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:59:24.961691 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.961844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961700    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.961844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961750    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.961844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961782    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-865qc\" (UniqueName: \"kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.961844 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.962068 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961883    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs\") pod \"d47c6373-6997-4b41-a561-4123456eefc5\" (UID: \"d47c6373-6997-4b41-a561-4123456eefc5\") "
Sep 08 18:59:24.962068 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.961994    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:24.962199 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.962179    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:24.962255 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.962184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home" (OuterVolumeSpecName: "home") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:24.962344 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.962272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:24.964509 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.964483    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm" (OuterVolumeSpecName: "dshm") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:24.964603 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.964533    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:24.964603 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.964547    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc" (OuterVolumeSpecName: "kube-api-access-865qc") pod "d47c6373-6997-4b41-a561-4123456eefc5" (UID: "d47c6373-6997-4b41-a561-4123456eefc5"). InnerVolumeSpecName "kube-api-access-865qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:59:24.974914 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.974881    2582 generic.go:356] "Generic (PLEG): container finished" podID="d47c6373-6997-4b41-a561-4123456eefc5" containerID="231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15" exitCode=0
Sep 08 18:59:24.975062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.974923    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"
Sep 08 18:59:24.975062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.974962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" event={"ID":"d47c6373-6997-4b41-a561-4123456eefc5","Type":"ContainerDied","Data":"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"}
Sep 08 18:59:24.975062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.975022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g" event={"ID":"d47c6373-6997-4b41-a561-4123456eefc5","Type":"ContainerDied","Data":"73327086d893eb49e352231400fd646b97a74d39079882c8fb9e7d3fb4d2d402"}
Sep 08 18:59:24.975062 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.975046    2582 scope.go:117] "RemoveContainer" containerID="231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"
Sep 08 18:59:24.984394 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.984374    2582 scope.go:117] "RemoveContainer" containerID="231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"
Sep 08 18:59:24.984745 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:59:24.984708    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15\": container with ID starting with 231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15 not found: ID does not exist" containerID="231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"
Sep 08 18:59:24.984808 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.984759    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15"} err="failed to get container status \"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15\": rpc error: code = NotFound desc = could not find container \"231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15\": container with ID starting with 231af8c49244692908b6294cc0b16b5ac7680ccb27a6e90c0f3a8eb33788ff15 not found: ID does not exist"
Sep 08 18:59:24.998745 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:24.998687    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:59:25.002240 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.002207    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-98edebc4-kserve-c864c7bf5-qhd6g"]
Sep 08 18:59:25.063072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.063037    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:25.063072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.063062    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:25.063072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.063073    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-865qc\" (UniqueName: \"kubernetes.io/projected/d47c6373-6997-4b41-a561-4123456eefc5-kube-api-access-865qc\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:25.063305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.063083    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47c6373-6997-4b41-a561-4123456eefc5-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:25.063305 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:25.063093    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47c6373-6997-4b41-a561-4123456eefc5-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.019287 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.019247    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" podUID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" containerName="main" containerID="cri-o://a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6" gracePeriod=45
Sep 08 18:59:26.098970 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.098936    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d47c6373-6997-4b41-a561-4123456eefc5" path="/var/lib/kubelet/pods/d47c6373-6997-4b41-a561-4123456eefc5/volumes"
Sep 08 18:59:26.276343 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.276283    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:59:26.372158 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372115    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4tjj\" (UniqueName: \"kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372184    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372260    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372333 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372318    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir\") pod \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\" (UID: \"e2d417ed-9d09-4c7f-b639-bc5d68aab78d\") "
Sep 08 18:59:26.372589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home" (OuterVolumeSpecName: "home") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:26.372589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372571    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:26.372712 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.372677    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:26.374475 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.374453    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm" (OuterVolumeSpecName: "dshm") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 18:59:26.374975 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.374952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:26.375071 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.374975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj" (OuterVolumeSpecName: "kube-api-access-q4tjj") pod "e2d417ed-9d09-4c7f-b639-bc5d68aab78d" (UID: "e2d417ed-9d09-4c7f-b639-bc5d68aab78d"). InnerVolumeSpecName "kube-api-access-q4tjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:59:26.473247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473215    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.473247 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473246    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.473451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473260    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.473451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473271    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.473451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473282    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.473451 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.473293    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4tjj\" (UniqueName: \"kubernetes.io/projected/e2d417ed-9d09-4c7f-b639-bc5d68aab78d-kube-api-access-q4tjj\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:26.986798 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.986758    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" containerID="a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6" exitCode=0
Sep 08 18:59:26.986978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.986810    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"
Sep 08 18:59:26.986978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.986841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" event={"ID":"e2d417ed-9d09-4c7f-b639-bc5d68aab78d","Type":"ContainerDied","Data":"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"}
Sep 08 18:59:26.986978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.986879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z" event={"ID":"e2d417ed-9d09-4c7f-b639-bc5d68aab78d","Type":"ContainerDied","Data":"c1bccb92a6cb7e91cb3b356260404f34082dfe3051ac85f37924ddd7a208cd2d"}
Sep 08 18:59:26.986978 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.986895    2582 scope.go:117] "RemoveContainer" containerID="a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"
Sep 08 18:59:26.996257 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.996228    2582 scope.go:117] "RemoveContainer" containerID="a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"
Sep 08 18:59:26.996531 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:59:26.996513    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6\": container with ID starting with a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6 not found: ID does not exist" containerID="a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"
Sep 08 18:59:26.996591 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:26.996541    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6"} err="failed to get container status \"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6\": rpc error: code = NotFound desc = could not find container \"a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6\": container with ID starting with a96f3f5e6b63b247df4e4a6f6868033c85b3f56f39f6488456b66024baad14d6 not found: ID does not exist"
Sep 08 18:59:27.008526 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.008499    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:59:27.022136 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022109    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:27.022503 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022488    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47c6373-6997-4b41-a561-4123456eefc5" containerName="main"
Sep 08 18:59:27.022562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022506    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47c6373-6997-4b41-a561-4123456eefc5" containerName="main"
Sep 08 18:59:27.022562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022536    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" containerName="main"
Sep 08 18:59:27.022562 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022544    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" containerName="main"
Sep 08 18:59:27.022677 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d47c6373-6997-4b41-a561-4123456eefc5" containerName="main"
Sep 08 18:59:27.022795 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.022695    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" containerName="main"
Sep 08 18:59:27.026790 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.026770    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-98edebc4-kserve-b6d6957f5-wdg8z"]
Sep 08 18:59:27.026926 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.026853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.030124 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.030106    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wvhnh\""
Sep 08 18:59:27.030124 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.030118    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 18:59:27.030293 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.030118    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 18:59:27.036269 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.036247    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:27.079999 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.079963    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.080162 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.080036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.080162 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.080070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.080162 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.080085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zw9\" (UniqueName: \"kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.181395 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.181357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.181597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.181452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.181597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.181508    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.181597 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.181526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zw9\" (UniqueName: \"kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.182138 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.182107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.184123 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.184101    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.184217 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.184149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.189361 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.189339    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zw9\" (UniqueName: \"kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9\") pod \"maas-api-dd87ddf4c-8zcp8\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") " pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.338002 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.337901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:27.612738 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.612692    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:27.615500 ip-10-0-142-80 kubenswrapper[2582]: W0908 18:59:27.615454    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3252da_2709_4feb_bef4_a6fc5718b91d.slice/crio-d0a99e6421bc20859d921eb4f11f439993bf544e8ecdf46166978276042e6001 WatchSource:0}: Error finding container d0a99e6421bc20859d921eb4f11f439993bf544e8ecdf46166978276042e6001: Status 404 returned error can't find the container with id d0a99e6421bc20859d921eb4f11f439993bf544e8ecdf46166978276042e6001
Sep 08 18:59:27.617066 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.617047    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 18:59:27.993430 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:27.993392    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" event={"ID":"ab3252da-2709-4feb-bef4-a6fc5718b91d","Type":"ContainerStarted","Data":"d0a99e6421bc20859d921eb4f11f439993bf544e8ecdf46166978276042e6001"}
Sep 08 18:59:28.099589 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:28.099544    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d417ed-9d09-4c7f-b639-bc5d68aab78d" path="/var/lib/kubelet/pods/e2d417ed-9d09-4c7f-b639-bc5d68aab78d/volumes"
Sep 08 18:59:30.009072 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:30.009035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" event={"ID":"ab3252da-2709-4feb-bef4-a6fc5718b91d","Type":"ContainerStarted","Data":"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"}
Sep 08 18:59:30.009478 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:30.009175    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:30.028538 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:30.028483    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" podStartSLOduration=1.288068774 podStartE2EDuration="3.028464239s" totalImagesPullingTime="1.740395465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:59:30.026561562 +0000 UTC m=+2144.478136698" watchObservedRunningTime="2026-09-08 18:59:30.028464239 +0000 UTC m=+2144.480039371"
Sep 08 18:59:34.485876 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.485835    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:34.486288 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.486038    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" podUID="ab3252da-2709-4feb-bef4-a6fc5718b91d" containerName="maas-api" containerID="cri-o://7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d" gracePeriod=30
Sep 08 18:59:34.491221 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.491196    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:34.734327 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.734299    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:34.845690 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.845599    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs\") pod \"ab3252da-2709-4feb-bef4-a6fc5718b91d\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") "
Sep 08 18:59:34.845690 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.845664    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls\") pod \"ab3252da-2709-4feb-bef4-a6fc5718b91d\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") "
Sep 08 18:59:34.845690 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.845683    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67zw9\" (UniqueName: \"kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9\") pod \"ab3252da-2709-4feb-bef4-a6fc5718b91d\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") "
Sep 08 18:59:34.846038 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.845764    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca\") pod \"ab3252da-2709-4feb-bef4-a6fc5718b91d\" (UID: \"ab3252da-2709-4feb-bef4-a6fc5718b91d\") "
Sep 08 18:59:34.846191 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.846167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab3252da-2709-4feb-bef4-a6fc5718b91d" (UID: "ab3252da-2709-4feb-bef4-a6fc5718b91d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 18:59:34.848041 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.848015    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab3252da-2709-4feb-bef4-a6fc5718b91d" (UID: "ab3252da-2709-4feb-bef4-a6fc5718b91d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:34.848143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.848071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab3252da-2709-4feb-bef4-a6fc5718b91d" (UID: "ab3252da-2709-4feb-bef4-a6fc5718b91d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 18:59:34.848143 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.848118    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9" (OuterVolumeSpecName: "kube-api-access-67zw9") pod "ab3252da-2709-4feb-bef4-a6fc5718b91d" (UID: "ab3252da-2709-4feb-bef4-a6fc5718b91d"). InnerVolumeSpecName "kube-api-access-67zw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 18:59:34.947397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.947357    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-metrics-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:34.947397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.947387    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab3252da-2709-4feb-bef4-a6fc5718b91d-maas-api-tls\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:34.947397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.947396    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zw9\" (UniqueName: \"kubernetes.io/projected/ab3252da-2709-4feb-bef4-a6fc5718b91d-kube-api-access-67zw9\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:34.947397 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:34.947404    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab3252da-2709-4feb-bef4-a6fc5718b91d-openshift-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 08 18:59:35.033101 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.033066    2582 generic.go:356] "Generic (PLEG): container finished" podID="ab3252da-2709-4feb-bef4-a6fc5718b91d" containerID="7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d" exitCode=0
Sep 08 18:59:35.033245 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.033112    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"
Sep 08 18:59:35.033245 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.033142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" event={"ID":"ab3252da-2709-4feb-bef4-a6fc5718b91d","Type":"ContainerDied","Data":"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"}
Sep 08 18:59:35.033245 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.033197    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8" event={"ID":"ab3252da-2709-4feb-bef4-a6fc5718b91d","Type":"ContainerDied","Data":"d0a99e6421bc20859d921eb4f11f439993bf544e8ecdf46166978276042e6001"}
Sep 08 18:59:35.033245 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.033221    2582 scope.go:117] "RemoveContainer" containerID="7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"
Sep 08 18:59:35.042539 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.042521    2582 scope.go:117] "RemoveContainer" containerID="7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"
Sep 08 18:59:35.042800 ip-10-0-142-80 kubenswrapper[2582]: E0908 18:59:35.042783    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d\": container with ID starting with 7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d not found: ID does not exist" containerID="7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"
Sep 08 18:59:35.042856 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.042808    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d"} err="failed to get container status \"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d\": rpc error: code = NotFound desc = could not find container \"7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d\": container with ID starting with 7420acc08f17e01d9063b05d9aee1a688930d5bd867bb9fc40363100dc9fef5d not found: ID does not exist"
Sep 08 18:59:35.055187 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.055160    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:35.059098 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:35.059070    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-dd87ddf4c-8zcp8"]
Sep 08 18:59:36.100303 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:36.100250    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab3252da-2709-4feb-bef4-a6fc5718b91d" path="/var/lib/kubelet/pods/ab3252da-2709-4feb-bef4-a6fc5718b91d/volumes"
Sep 08 18:59:46.579376 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:46.579283    2582 scope.go:117] "RemoveContainer" containerID="6ed2ee858251f4115d723385f91beecaee92d65558b56990ae8f0e8186556374"
Sep 08 18:59:46.589050 ip-10-0-142-80 kubenswrapper[2582]: I0908 18:59:46.589028    2582 scope.go:117] "RemoveContainer" containerID="b8fe2274773ef72deef2676af893af834716bfc99ccb572ead58022c44d74268"
Sep 08 19:00:00.139991 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.139949    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"]
Sep 08 19:00:00.140483 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.140460    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab3252da-2709-4feb-bef4-a6fc5718b91d" containerName="maas-api"
Sep 08 19:00:00.140558 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.140487    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab3252da-2709-4feb-bef4-a6fc5718b91d" containerName="maas-api"
Sep 08 19:00:00.140623 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.140593    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab3252da-2709-4feb-bef4-a6fc5718b91d" containerName="maas-api"
Sep 08 19:00:00.143600 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.143577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"
Sep 08 19:00:00.146569 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.146545    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g6q8h\""
Sep 08 19:00:00.152445 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.152422    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"]
Sep 08 19:00:00.154218 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.154199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"
Sep 08 19:00:00.424669 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:00.424646    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"]
Sep 08 19:00:00.427052 ip-10-0-142-80 kubenswrapper[2582]: W0908 19:00:00.427023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09def42f_8c06_4e04_b485_721019dbd749.slice/crio-56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1 WatchSource:0}: Error finding container 56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1: Status 404 returned error can't find the container with id 56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1
Sep 08 19:00:01.152878 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:01.152839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66" event={"ID":"09def42f-8c06-4e04-b485-721019dbd749","Type":"ContainerStarted","Data":"75200145e9cfd3054221319256cdf9ba36e915ebd8346a75319da660c136516c"}
Sep 08 19:00:01.153275 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:01.152909    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66" event={"ID":"09def42f-8c06-4e04-b485-721019dbd749","Type":"ContainerStarted","Data":"56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1"}
Sep 08 19:00:01.169009 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:01.168963    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66" podStartSLOduration=1.168948181 podStartE2EDuration="1.168948181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:01.168835669 +0000 UTC m=+2175.620410800" watchObservedRunningTime="2026-09-08 19:00:01.168948181 +0000 UTC m=+2175.620523350"
Sep 08 19:00:02.159874 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.159836    2582 generic.go:356] "Generic (PLEG): container finished" podID="09def42f-8c06-4e04-b485-721019dbd749" containerID="75200145e9cfd3054221319256cdf9ba36e915ebd8346a75319da660c136516c" exitCode=0
Sep 08 19:00:02.160366 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.159939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66" event={"ID":"09def42f-8c06-4e04-b485-721019dbd749","Type":"ContainerDied","Data":"75200145e9cfd3054221319256cdf9ba36e915ebd8346a75319da660c136516c"}
Sep 08 19:00:02.245346 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.245322    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xctwd_00bd7b67-4acc-41ae-bc51-4510fc333952/manager/0.log"
Sep 08 19:00:02.358496 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.358466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s4j9j_127b620a-72bc-4954-ae6a-b8fe38eb0b77/manager/0.log"
Sep 08 19:00:02.618943 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.618867    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5kcz6_5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c/server/0.log"
Sep 08 19:00:02.734075 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.734043    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rwjb_29dc83f2-268a-4a5f-8f80-fa4d83b6141c/manager/0.log"
Sep 08 19:00:02.844199 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.844171    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-6ccqg_9ec45c56-cd0e-4e44-82af-361d05980a6b/copy-manifests/0.log"
Sep 08 19:00:02.851315 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:02.851292    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-6ccqg_9ec45c56-cd0e-4e44-82af-361d05980a6b/manager/0.log"
Sep 08 19:00:03.307920 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:03.307891    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"
Sep 08 19:00:03.675905 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:03.675882    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814885-cllpr_b7d8d1f6-b3cd-4d49-ba7f-39eebc765b86/cleanup/0.log"
Sep 08 19:00:03.785548 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:03.785518    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814900-rxx66_09def42f-8c06-4e04-b485-721019dbd749/cleanup/0.log"
Sep 08 19:00:03.899735 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:03.899676    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2h7f4_fa33bc29-0138-4074-b22d-4b34e3b694c5/postgres/0.log"
Sep 08 19:00:04.170679 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:04.170646    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66"
Sep 08 19:00:04.170869 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:04.170646    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-rxx66" event={"ID":"09def42f-8c06-4e04-b485-721019dbd749","Type":"ContainerDied","Data":"56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1"}
Sep 08 19:00:04.170869 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:04.170779    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e0eae2e160ee596cf652f6b3f68862f7b2b96ab174267f7cd4cc4670bddcb1"
Sep 08 19:00:04.844879 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:04.844849    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-ltp7b_e7c13d54-944a-463e-aa56-98dc106c4d8d/manager/0.log"
Sep 08 19:00:05.451040 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:05.451009    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-snrsq_d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30/manager/0.log"
Sep 08 19:00:05.795107 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:05.795023    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x_53d5926a-58dc-4de1-859e-3cde04f4585c/istio-proxy/0.log"
Sep 08 19:00:06.036920 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:06.036890    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d985cc74d-9hb2s_caf80ae4-f7dd-44ec-ab5c-1613a234f114/kube-auth-proxy/0.log"
Sep 08 19:00:06.294182 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:06.294134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/1.log"
Sep 08 19:00:06.412147 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:06.412105    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hrsx4_31c80b8a-a736-4f60-88f8-c80e53c864ec/payload-pre-processing/0.log"
Sep 08 19:00:06.537587 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:06.537562    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 19:00:07.351719 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:07.351689    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-m6dfl_6242c40e-0f64-4a81-a7ae-a75d62db53e3/main/0.log"
Sep 08 19:00:07.578166 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:07.578133    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lbjbk_1753665c-0bd5-4c47-83d3-bec38f004adb/main/0.log"
Sep 08 19:00:13.781391 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.781355    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phw82/must-gather-5bgkb"]
Sep 08 19:00:13.781904 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.781885    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09def42f-8c06-4e04-b485-721019dbd749" containerName="cleanup"
Sep 08 19:00:13.781983 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.781907    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09def42f-8c06-4e04-b485-721019dbd749" containerName="cleanup"
Sep 08 19:00:13.782042 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.781992    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09def42f-8c06-4e04-b485-721019dbd749" containerName="cleanup"
Sep 08 19:00:13.785973 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.785951    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.788954 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.788922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phw82\"/\"openshift-service-ca.crt\""
Sep 08 19:00:13.789068 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.788933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phw82\"/\"kube-root-ca.crt\""
Sep 08 19:00:13.790230 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.790209    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-phw82\"/\"default-dockercfg-fm6tw\""
Sep 08 19:00:13.794019 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.793992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/must-gather-5bgkb"]
Sep 08 19:00:13.875008 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.874976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22s6\" (UniqueName: \"kubernetes.io/projected/59cab635-baba-4527-8e3b-c2e2f55b4f6c-kube-api-access-b22s6\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.875190 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.875054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59cab635-baba-4527-8e3b-c2e2f55b4f6c-must-gather-output\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.975577 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.975536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22s6\" (UniqueName: \"kubernetes.io/projected/59cab635-baba-4527-8e3b-c2e2f55b4f6c-kube-api-access-b22s6\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.975762 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.975618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59cab635-baba-4527-8e3b-c2e2f55b4f6c-must-gather-output\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.975982 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.975964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59cab635-baba-4527-8e3b-c2e2f55b4f6c-must-gather-output\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:13.985895 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:13.985866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22s6\" (UniqueName: \"kubernetes.io/projected/59cab635-baba-4527-8e3b-c2e2f55b4f6c-kube-api-access-b22s6\") pod \"must-gather-5bgkb\" (UID: \"59cab635-baba-4527-8e3b-c2e2f55b4f6c\") " pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:14.095372 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:14.095284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/must-gather-5bgkb"
Sep 08 19:00:14.367538 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:14.367418    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/must-gather-5bgkb"]
Sep 08 19:00:14.370137 ip-10-0-142-80 kubenswrapper[2582]: W0908 19:00:14.370110    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59cab635_baba_4527_8e3b_c2e2f55b4f6c.slice/crio-fdcafe9361bc4dbb018b9b7f271ed11ba9a3064faa55f4cf481d69576f0abe29 WatchSource:0}: Error finding container fdcafe9361bc4dbb018b9b7f271ed11ba9a3064faa55f4cf481d69576f0abe29: Status 404 returned error can't find the container with id fdcafe9361bc4dbb018b9b7f271ed11ba9a3064faa55f4cf481d69576f0abe29
Sep 08 19:00:15.220344 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:15.220318    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/must-gather-5bgkb" event={"ID":"59cab635-baba-4527-8e3b-c2e2f55b4f6c","Type":"ContainerStarted","Data":"fdcafe9361bc4dbb018b9b7f271ed11ba9a3064faa55f4cf481d69576f0abe29"}
Sep 08 19:00:16.228808 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:16.228693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/must-gather-5bgkb" event={"ID":"59cab635-baba-4527-8e3b-c2e2f55b4f6c","Type":"ContainerStarted","Data":"8452af1c33ba90a1c5e8c7b0f7cdc70b5ab506f5aae6e639b21e9e07ede8f7d9"}
Sep 08 19:00:16.228808 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:16.228779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/must-gather-5bgkb" event={"ID":"59cab635-baba-4527-8e3b-c2e2f55b4f6c","Type":"ContainerStarted","Data":"e5d43d3d654faa2428b6540894a52dfb645c73eb48ac078b56a5207f7ec6f084"}
Sep 08 19:00:16.246043 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:16.245980    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-phw82/must-gather-5bgkb" podStartSLOduration=2.402956266 podStartE2EDuration="3.245957702s" totalImagesPullingTime="843.001436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:16.244405167 +0000 UTC m=+2190.695980300" watchObservedRunningTime="2026-09-08 19:00:16.245957702 +0000 UTC m=+2190.697532836"
Sep 08 19:00:16.887795 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:16.887757    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sb25t_6e00ad94-0553-41e4-adf2-d9c5971cc0ab/global-pull-secret-syncer/0.log"
Sep 08 19:00:16.987317 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:16.987281    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pbmrz_2ebeebd7-59c9-428f-bb26-025899cc0bb8/konnectivity-agent/0.log"
Sep 08 19:00:17.110926 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:17.110884    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-80.ec2.internal_9b7a1699332bcb5df721e142c96ab97c/haproxy/0.log"
Sep 08 19:00:21.980987 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:21.980949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-ltp7b_e7c13d54-944a-463e-aa56-98dc106c4d8d/manager/0.log"
Sep 08 19:00:22.247806 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:22.247293    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-snrsq_d818f92f-c8f3-4c2c-9f2e-d8c1c2a79f30/manager/0.log"
Sep 08 19:00:23.985625 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:23.985567    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72rsq_e9a26264-f9d3-4ce6-945b-81ab8e926f17/node-exporter/0.log"
Sep 08 19:00:24.017747 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:24.017695    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72rsq_e9a26264-f9d3-4ce6-945b-81ab8e926f17/kube-rbac-proxy/0.log"
Sep 08 19:00:24.045964 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:24.045931    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-72rsq_e9a26264-f9d3-4ce6-945b-81ab8e926f17/init-textfile/0.log"
Sep 08 19:00:27.035389 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.035345    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"]
Sep 08 19:00:27.043674 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.043641    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.048333 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.048305    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"]
Sep 08 19:00:27.108503 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.108466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-podres\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.108711 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.108515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-sys\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.108711 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.108581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j82kq\" (UniqueName: \"kubernetes.io/projected/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-kube-api-access-j82kq\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.108711 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.108612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-proc\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.108711 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.108649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-lib-modules\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.209847 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.209811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-proc\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210069 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.209863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-lib-modules\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210069 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.209893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-podres\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210069 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.209926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-sys\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210069 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.209976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j82kq\" (UniqueName: \"kubernetes.io/projected/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-kube-api-access-j82kq\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210069 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.210048    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-lib-modules\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210375 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.210130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-sys\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210375 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.210201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-podres\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.210613 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.210583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-proc\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.220975 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.220944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j82kq\" (UniqueName: \"kubernetes.io/projected/d0ba0dec-67e6-4db6-85c6-a68281a3ab3c-kube-api-access-j82kq\") pod \"perf-node-gather-daemonset-6fgg6\" (UID: \"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c\") " pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.358617 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.358341    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:27.672464 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:27.672352    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"]
Sep 08 19:00:27.675469 ip-10-0-142-80 kubenswrapper[2582]: W0908 19:00:27.675419    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0ba0dec_67e6_4db6_85c6_a68281a3ab3c.slice/crio-9903654a0d370bc9498d6551e022379266d95bc9b2dae963bab2d75f38f6121c WatchSource:0}: Error finding container 9903654a0d370bc9498d6551e022379266d95bc9b2dae963bab2d75f38f6121c: Status 404 returned error can't find the container with id 9903654a0d370bc9498d6551e022379266d95bc9b2dae963bab2d75f38f6121c
Sep 08 19:00:28.314898 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:28.314855    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6" event={"ID":"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c","Type":"ContainerStarted","Data":"bc3ead5c4c77111182c17593019954c9b049d4d4bb392e5961e1241b8b4aa0ac"}
Sep 08 19:00:28.315365 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:28.314924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6" event={"ID":"d0ba0dec-67e6-4db6-85c6-a68281a3ab3c","Type":"ContainerStarted","Data":"9903654a0d370bc9498d6551e022379266d95bc9b2dae963bab2d75f38f6121c"}
Sep 08 19:00:28.332765 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:28.332685    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6" podStartSLOduration=1.332655406 podStartE2EDuration="1.332655406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 19:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:28.331042465 +0000 UTC m=+2202.782617598" watchObservedRunningTime="2026-09-08 19:00:28.332655406 +0000 UTC m=+2202.784230540"
Sep 08 19:00:29.288429 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:29.288396    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66zzj_b129a20f-63a5-46ff-94f2-73542d655a22/dns/0.log"
Sep 08 19:00:29.310358 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:29.310330    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-66zzj_b129a20f-63a5-46ff-94f2-73542d655a22/kube-rbac-proxy/0.log"
Sep 08 19:00:29.320952 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:29.320917    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:29.442298 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:29.442268    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qwcwx_93a88fc2-f535-4304-ae16-cd66d5924739/dns-node-resolver/0.log"
Sep 08 19:00:30.045491 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:30.045444    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kf9x2_62dbe9f7-1c35-42c3-be25-65e52c6409c4/node-ca/0.log"
Sep 08 19:00:31.003976 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.003939    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x_53d5926a-58dc-4de1-859e-3cde04f4585c/istio-proxy/0.log"
Sep 08 19:00:31.454337 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.454310    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d985cc74d-9hb2s_caf80ae4-f7dd-44ec-ab5c-1613a234f114/kube-auth-proxy/0.log"
Sep 08 19:00:31.514896 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.514854    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/0.log"
Sep 08 19:00:31.525008 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.524983    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/1.log"
Sep 08 19:00:31.566238 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.566207    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hrsx4_31c80b8a-a736-4f60-88f8-c80e53c864ec/payload-pre-processing/0.log"
Sep 08 19:00:31.606282 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:31.606252    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 19:00:32.582857 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:32.582819    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjwrf_c0861cf4-0c21-4246-b7b6-d36e6a654a02/serve-healthcheck-canary/0.log"
Sep 08 19:00:33.094274 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:33.094244    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bj964_98277700-ec42-487d-8021-e7ce8c97a640/kube-rbac-proxy/0.log"
Sep 08 19:00:33.118085 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:33.118060    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bj964_98277700-ec42-487d-8021-e7ce8c97a640/exporter/0.log"
Sep 08 19:00:33.144770 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:33.144741    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bj964_98277700-ec42-487d-8021-e7ce8c97a640/extractor/0.log"
Sep 08 19:00:35.341296 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.341264    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-phw82/perf-node-gather-daemonset-6fgg6"
Sep 08 19:00:35.387817 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.387787    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xctwd_00bd7b67-4acc-41ae-bc51-4510fc333952/manager/0.log"
Sep 08 19:00:35.616990 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.616909    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s4j9j_127b620a-72bc-4954-ae6a-b8fe38eb0b77/manager/0.log"
Sep 08 19:00:35.706339 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.706305    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5kcz6_5d58d9bc-d281-4cae-a9b2-1cbd07f1d25c/server/0.log"
Sep 08 19:00:35.809845 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.809809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9rwjb_29dc83f2-268a-4a5f-8f80-fa4d83b6141c/manager/0.log"
Sep 08 19:00:35.838535 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.838508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-6ccqg_9ec45c56-cd0e-4e44-82af-361d05980a6b/manager/0.log"
Sep 08 19:00:35.861559 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:35.861525    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8d688d88f-6ccqg_9ec45c56-cd0e-4e44-82af-361d05980a6b/copy-manifests/0.log"
Sep 08 19:00:37.423184 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:37.423144    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-v4ddv_20e7e43a-a0f9-4a7f-8cde-a5cab68927d9/openshift-lws-operator/0.log"
Sep 08 19:00:37.505501 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:37.505383    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x_53d5926a-58dc-4de1-859e-3cde04f4585c/istio-proxy/0.log"
Sep 08 19:00:37.951809 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:37.951761    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d985cc74d-9hb2s_caf80ae4-f7dd-44ec-ab5c-1613a234f114/kube-auth-proxy/0.log"
Sep 08 19:00:37.988478 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:37.988421    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/0.log"
Sep 08 19:00:38.000854 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:38.000820    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/1.log"
Sep 08 19:00:38.031754 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:38.031419    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hrsx4_31c80b8a-a736-4f60-88f8-c80e53c864ec/payload-pre-processing/0.log"
Sep 08 19:00:38.066752 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:38.066706    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 19:00:39.581336 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:39.581296    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-phc6x_53d5926a-58dc-4de1-859e-3cde04f4585c/istio-proxy/0.log"
Sep 08 19:00:40.017766 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:40.017704    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d985cc74d-9hb2s_caf80ae4-f7dd-44ec-ab5c-1613a234f114/kube-auth-proxy/0.log"
Sep 08 19:00:40.063277 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:40.063228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/0.log"
Sep 08 19:00:40.072140 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:40.072096    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tmgsr_245faeeb-9478-415a-b635-cfbee2b8cdd2/istio-proxy/1.log"
Sep 08 19:00:40.103003 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:40.102920    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-hrsx4_31c80b8a-a736-4f60-88f8-c80e53c864ec/payload-pre-processing/0.log"
Sep 08 19:00:40.134282 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:40.134225    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2lwjs_575a15a2-860b-40d6-b80e-1e9849564dab/payload-processing/0.log"
Sep 08 19:00:42.602426 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:42.602393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ghlrf_25b71bdb-9747-453b-a23d-a6c11b8ad390/migrator/0.log"
Sep 08 19:00:42.625623 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:42.625594    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ghlrf_25b71bdb-9747-453b-a23d-a6c11b8ad390/graceful-termination/0.log"
Sep 08 19:00:44.156999 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.156973    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/kube-multus-additional-cni-plugins/0.log"
Sep 08 19:00:44.180068 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.180043    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/egress-router-binary-copy/0.log"
Sep 08 19:00:44.208402 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.208377    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/cni-plugins/0.log"
Sep 08 19:00:44.232605 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.232577    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/bond-cni-plugin/0.log"
Sep 08 19:00:44.255776 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.255718    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/routeoverride-cni/0.log"
Sep 08 19:00:44.280013 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.279991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/whereabouts-cni-bincopy/0.log"
Sep 08 19:00:44.308132 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.308102    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8r4mg_6c7acc60-35eb-4357-bc47-5bfed8c77926/whereabouts-cni/0.log"
Sep 08 19:00:44.745437 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.745404    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hwhsw_3dce7655-4f24-49ff-a2b4-85a44d40bf5f/kube-multus/0.log"
Sep 08 19:00:44.774835 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.774807    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9br6q_623bbd40-a928-4099-9d04-964fa3145559/network-metrics-daemon/0.log"
Sep 08 19:00:44.792778 ip-10-0-142-80 kubenswrapper[2582]: I0908 19:00:44.792751    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9br6q_623bbd40-a928-4099-9d04-964fa3145559/kube-rbac-proxy/0.log"